Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1321524 > unrolled thread

Re: [PATCH] zram: export the number of available comp streams

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [PATCH] zram: export the number of available comp streams Minchan Kim <minchan@kernel.org> - 2016-01-29 08:30 +0100
    Re: [PATCH] zram: export the number of available comp streams Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-02-01 02:10 +0100

#1321524 — Re: [PATCH] zram: export the number of available comp streams

Hello Sergey,

Sorry to late response. Thesedays, I'm really busy with personal
stuff.

Today, I have thought about this patch and have several questions.
Let's start with simple questions.

On Tue, Jan 26, 2016 at 09:03:59PM +0900, Sergey Senozhatsky wrote:
> I've been asked several very simple questions:
> a) How can I ensure that zram uses (or used) several compression
>    streams?

Why does he want to ensure several compression streams?
As you know well, zram handle it dynamically.

If zram cannot allocate more streams, it means the system is
heavily fragmented or memory pressure at that time so there
is no worth to add more stream, I think.

Could you elaborate it more why he want to know it and what
he expect from that?

> b) What is the current number of comp streams (how much memory
>    does zram *actually* use for compression streams, if there are
>    more than one stream)?

Hmm, in the kernel, there are lots of example subsystem
we cannot know exact memory usage. Why does the user want
to know exact memory usage of zram? What is his concern?

In advance, sorry for slow response.

> 
> zram, indeed, does not provide any info and does not answer
> these questions. Reading from `max_comp_streams' let to estimate
> only theoretical comp streams memory consumption, which assumes
> that zram will allocate max_comp_streams. However, it's possible
> that the real number of compression streams will never reach that
> max value, due to various reasons, e.g. max_comp_streams is too
> high, etc.
> 
> The patch adds `avail_streams' column to the /sys/block/zram<id>/mm_stat
> device file. For a single compression stream backend it's always 1,
> for a multi stream backend - it shows the actual ->avail_strm value.
> 
> The number of allocated compression streams answers several
> questions:
> a) the current `level of concurrency' that the device has
>    experienced
> b) the amount of memory used by compression streams (by multiplying
>    the `avail_streams' column value, ->buffer size and algorithm's
>    specific scratch buffer size; the last are easy to find out,
>    unlike `avail_streams').
> 
> Signed-off-by: Sergey Senozhatsky <sergey.senozhatsky@gmail.com>

[toc] | [next] | [standalone]

#1322772

Hello Minchan,

On (01/29/16 16:28), Minchan Kim wrote:
> Hello Sergey,
> 
> Sorry to late response. Thesedays, I'm really busy with personal
> stuff.

sure, no worries :)

> On Tue, Jan 26, 2016 at 09:03:59PM +0900, Sergey Senozhatsky wrote:
> > I've been asked several very simple questions:
> > a) How can I ensure that zram uses (or used) several compression
> >    streams?
> 
> Why does he want to ensure several compression streams?
> As you know well, zram handle it dynamically.
> 
> If zram cannot allocate more streams, it means the system is
> heavily fragmented or memory pressure at that time so there
> is no worth to add more stream, I think.
> 
> Could you elaborate it more why he want to know it and what
> he expect from that?

good questions. I believe mostly it's about fine-tuning on a
per-device basis, which is getting especially tricky when zram
devices are used as a sort of in-memory tmp storage for various
applications (black boxen).

> > b) What is the current number of comp streams (how much memory
> >    does zram *actually* use for compression streams, if there are
> >    more than one stream)?
> 
> Hmm, in the kernel, there are lots of example subsystem
> we cannot know exact memory usage. Why does the user want
> to know exact memory usage of zram? What is his concern?

certainly true. probably some of those sub-systems/drivers have some
sort of LRU, or shrinker callbacks, to release unneeded memory back.
zram only allocates streams, and it basically hard to tell how many:
up to max_comp_streams, which can be larger than the number of cpus
on the system; because we keep preemption enabled (I didn't realize
that until I played with the patch) around
zcomp_strm_find()/zcomp_strm_release():

	zram_bvec_write()
	{
		...
		zstrm = zcomp_strm_find(zram->comp);
>> can preempt
		user_mem = kmap_atomic(page);
>> now atomic
		zcomp_compress()
		...
		kunmap_atomic()
>> can preempt
		zcomp_strm_release()
		...
	}

so how many streams I can have on my old 4-cpus x86_64 box?

10?
yes.

# cat /sys/block/zram0/mm_stat
630484992  9288707 13103104        0 13103104    16240        0       10

16?
yes.

# cat /sys/block/zram0/mm_stat
1893117952 25296718 31354880        0 31354880    15342        0       16

21?
yes.

# cat /sys/block/zram0/mm_stat
1893167104 28499936 46616576        0 46616576    15330        0       21

do I need 21? may be no. do I nede 18? if 18 streams are needed only 10%
of the time (I can figure it out by doing repetitive cat zramX/mm_stat),
then I can set max_comp_streams to make 90% of applications happy, e.g.
max_comp_streams to 10, and save some memory.

10 echo X > /sys/block/zramX/max_comp_streams
20 do tests
30 cat /sys/block/zramX/mm_stat
40 update X (increase/decrease) if needed, otherwise break
50 goto 10


> In advance, sorry for slow response.

no prob, my response wasn't super fast either :)

	-ss

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web