Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]
Groups > linux.kernel > #1363235 > unrolled thread
| Started by | Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> |
|---|---|
| First post | 2016-03-23 09:20 +0100 |
| Last post | 2016-04-01 17:50 +0200 |
| Articles | 10 — 3 participants |
Back to article view | Back to linux.kernel
Re: zram: per-cpu compression streams Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-03-23 09:20 +0100
Re: zram: per-cpu compression streams Minchan Kim <minchan@kernel.org> - 2016-03-25 00:50 +0100
Re: zram: per-cpu compression streams Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-03-25 02:50 +0100
Re: zram: per-cpu compression streams Minchan Kim <minchan@kernel.org> - 2016-03-28 05:30 +0200
Re: zram: per-cpu compression streams Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-03-30 10:40 +0200
Re: zram: per-cpu compression streams Minchan Kim <minchan@kernel.org> - 2016-03-31 00:20 +0200
Re: zram: per-cpu compression streams Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-03-31 03:30 +0200
Re: zram: per-cpu compression streams Minchan Kim <minchan@kernel.org> - 2016-03-31 08:00 +0200
Re: zram: per-cpu compression streams Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> - 2016-03-31 08:40 +0200
Re: zram: per-cpu compression streams Sergey Senozhatsky <sergey.senozhatsky@gmail.com> - 2016-04-01 17:50 +0200
| From | Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> |
|---|---|
| Date | 2016-03-23 09:20 +0100 |
| Subject | Re: zram: per-cpu compression streams |
| Message-ID | <rfMbg-CU-1@gated-at.bofh.it> |
( was "[PATCH] zram: export the number of available comp streams"
forked from http://marc.info/?l=linux-kernel&m=145860707516861 )
d'oh.... sorry, now actually forked.
Hello Minchan,
forked into a separate tread.
> On (03/22/16 09:39), Minchan Kim wrote:
> > zram_bvec_write()
> > {
> > *get_cpu_ptr(comp-stream);
> > zcomp_compress();
> > zs_malloc()
> > put_cpu_ptr(comp-stream);
> > }
> >
> > this, however, makes zsmalloc unhapy. pool has GFP_NOIO | __GFP_HIGHMEM
> > gfp, and GFP_NOIO is ___GFP_DIRECT_RECLAIM|___GFP_KSWAPD_RECLAIM. this
> > __GFP_DIRECT_RECLAIM is in the conflict with per-cpu streams, because
> > per-cpu streams require disabled preemption (up until we copy stream
> > buffer to zspage). so what options do we have here... from the top of
> > my head (w/o a lot of thinking)...
>
> Indeed.
...
> How about this?
>
> zram_bvec_write()
> {
> retry:
> *get_cpu_ptr(comp-stream);
> zcomp_compress();
> handle = zs_malloc((gfp &~ __GFP_DIRECT_RECLAIM| | GFP_NOWARN)
> if (!handle) {
> put_cpu_ptr(comp-stream);
> handle = zs_malloc(gfp);
> goto retry;
> }
> put_cpu_ptr(comp-stream);
> }
interesting. the retry jump should go higher, we have "user_mem = kmap_atomic(page)"
which we unmap right after compression, because a) we don't need
uncompressed memory anymore b) zs_malloc() can sleep and we can't have atomic
mapping around. the nasty thing here is is_partial_io(). we need to re-do
if (is_partial_io(bvec))
memcpy(uncmem + offset, user_mem + bvec-bv_offset,
bvec-bv_len);
once again in the worst case.
so zs_malloc((gfp &~ __GFP_DIRECT_RECLAIM | GFP_NOWARN) so far can cause
double memcpy() and double compression. just to outline this.
the test.
I executed a number of iozone tests, on each iteration re-creating zram
device (3GB, LZO, EXT4. the box has 4 x86_64 CPUs).
$DEVICE_SZ=3G
$FREE_SPACE is 10% of $DEVICE_SZ
time ./iozone -t $i -R -r $((8*$i))K -s $((($DEVICE_SZ/$i - $FREE_SPACE)/(1024*1024)))M -I +Z
columns:
TEST MAX_STREAMS 4 MAX_STREAMS 8 PER_CPU STREAMS
====================================================================
Test #1 iozone -t 1 -R -r 8K -s 2764M -I +Z
Initial write 853492.31* 835868.50 839789.56
Rewrite 1642073.88 1657255.75 1693011.50*
Read 3384044.00* 3218727.25 3269109.50
Re-read 3389794.50* 3243187.00 3267422.25
Reverse Read 3209805.75* 3082040.00 3107957.25
Stride read 3100144.50* 2972280.25 2923155.25
Random read 2992249.75* 2874605.00 2854824.25
Mixed workload 2992274.75* 2878212.25 2883840.00
Random write 1471800.00 1452346.50 1515678.75*
Pwrite 802083.00 801627.31 820251.69*
Pread 3443495.00* 3308659.25 3302089.00
Fwrite 1880446.88 1838607.50 1909490.00*
Fread 3479614.75 3091634.75 6442964.50*
= real 1m4.170s 1m4.513s 1m4.123s
= user 0m0.559s 0m0.518s 0m0.511s
= sys 0m18.766s 0m19.264s 0m18.641s
Test #2 iozone -t 2 -R -r 16K -s 1228M -I +Z
Initial write 2102532.12 2051809.19 2419072.50*
Rewrite 2217024.25 2250930.00 3681559.00*
Read 7716933.25 7898759.00 8345507.75*
Re-read 7748487.75 7765282.25 8342367.50*
Reverse Read 7415254.25 7552637.25 7822691.75*
Stride read 7041909.50 7091049.25 7401273.00*
Random read 6205044.25 6738888.50 7232104.25*
Mixed workload 4582990.00 5271651.50 5361002.88*
Random write 2591893.62 2513729.88 3660774.38*
Pwrite 1873876.75 1909758.69 2087238.81*
Pread 4669850.00 4651121.56 4919588.44*
Fwrite 1937947.25 1940628.06 2034251.25*
Fread 9930319.00 9970078.00* 9831422.50
= real 0m53.844s 0m53.607s 0m52.528s
= user 0m0.273s 0m0.289s 0m0.280s
= sys 0m16.595s 0m16.478s 0m14.072s
Test #3 iozone -t 3 -R -r 24K -s 716M -I +Z
Initial write 3036567.50 2998918.25 3683853.00*
Rewrite 3402447.88 3415685.88 5054705.38*
Read 11767413.00* 11133789.50 11246497.25
Re-read 11797680.50* 11092592.00 11277382.00
Reverse Read 10828320.00* 10157665.50 10749055.00
Stride read 10532039.50* 9943521.75 10464700.25
Random read 10380365.75* 9807859.25 10234127.00
Mixed workload 8772132.50* 8415083.50 8457108.50
Random write 3364875.00 3310042.00 5059136.38*
Pwrite 2677290.25 2651309.50 3198166.25*
Pread 5221799.56* 4963050.69 4987293.78
Fwrite 2026887.56 2047679.00 2124199.62*
Fread 11310381.25 11413531.50 11444208.75*
= real 0m50.209s 0m50.782s 0m49.750s
= user 0m0.195s 0m0.205s 0m0.215s
= sys 0m14.873s 0m15.159s 0m12.911s
Test #4 iozone -t 4 -R -r 32K -s 460M -I +Z
Initial write 3841474.94 3859279.81 5309988.88*
Rewrite 3905526.25 3917309.62 6814800.62*
Read 16233054.50 14843560.25 16352283.75*
Re-read 16335506.50 15529152.25 16352570.00*
Reverse Read 15316394.50* 14225482.50 15004897.50
Stride read 14799380.25* 14064034.25 14355184.25
Random read 14683771.00 14206928.50 14814913.00*
Mixed workload 9058851.50 9180650.75 10815917.50*
Random write 3990585.94 4004757.00 6722088.50*
Pwrite 3318836.12 3468977.69 4244747.69*
Pread 5894538.16* 5588046.38 5847345.62
Fwrite 2227353.75 2186688.62 2386974.88*
Fread 12046094.00 12240004.75* 12073956.75
= real 0m48.561s 0m48.839s 0m48.142s
= user 0m0.155s 0m0.170s 0m0.133s
= sys 0m13.650s 0m13.684s 0m10.790s
Test #5 iozone -t 5 -R -r 40K -s 307M -I +Z
Initial write 4034878.94 4026610.69 5775746.12*
Rewrite 3898600.44 3901114.16 6923764.19*
Read 14947360.88 16698824.25* 10155333.62
Re-read 15844580.75* 15344057.00 9869874.38
Reverse Read 7459156.95 9023317.86* 7648295.03
Stride read 10823891.81 9615553.81 11231183.72*
Random read 10391702.56* 9740935.75 10048038.28
Mixed workload 8261830.94 10175925.00* 7535763.75
Random write 3951423.31 3960984.62 6671441.38*
Pwrite 4119023.12 4097204.56 5975659.12*
Pread 6072076.73* 4338668.50 6020808.34
Fwrite 2417235.47 2337875.88 2665450.62*
Fread 13393630.25 13648332.00* 13395391.00
= real 0m47.756s 0m47.939s 0m47.483s
= user 0m0.128s 0m0.128s 0m0.119s
= sys 0m10.361s 0m10.392s 0m8.717s
Test #6 iozone -t 6 -R -r 48K -s 204M -I +Z
Initial write 4134932.97 4137171.88 5983193.31*
Rewrite 3928131.31 3950764.00 7124248.00*
Read 10965005.75* 10152236.50 9856572.88
Re-read 9386946.00 10776231.38 14303174.12*
Reverse Read 6035244.89 7456152.38* 5999446.38
Stride read 8041000.75 7995307.75 10182936.75*
Random read 8565099.09 10487707.58* 8694877.25
Mixed workload 5301593.06 7332589.09* 6802251.06
Random write 4046482.56 3986854.94 6723824.56*
Pwrite 4188226.41 4214513.34 6245278.44*
Pread 3452596.86 3708694.69* 3486420.41
Fwrite 2829500.22 3030742.72 3033792.28*
Fread 13331387.75 13490416.50 14940410.25*
= real 0m47.150s 0m47.050s 0m47.044s
= user 0m0.106s 0m0.100s 0m0.094s
= sys 0m9.238s 0m8.804s 0m6.930s
Test #7 iozone -t 7 -R -r 56K -s 131M -I +Z
Initial write 4169480.84 4116331.03 5946801.38*
Rewrite 3993155.97 3986195.00 6928142.44*
Read 18901600.25* 10088918.69 6699592.78
Re-read 8738544.69 14881309.62* 13960026.06
Reverse Read 5008919.08 7923949.95* 5495212.41
Stride read 7029436.75 8747574.91* 6477087.25
Random read 6994738.56* 5448687.81 6585235.53
Mixed workload 5178632.44 5258914.92 5587421.81*
Random write 4008977.78 3928116.88 6816453.12*
Pwrite 4342852.09 4154319.09 6124520.06*
Pread 3880318.99 2978587.56 4493903.14*
Fwrite 5557990.03 2923556.59 6126649.94*
Fread 14451722.00 15281179.62* 14675436.50
= real 0m46.321s 0m46.458s 0m45.791s
= user 0m0.093s 0m0.089s 0m0.095s
= sys 0m6.961s 0m6.600s 0m5.499s
Test #8 iozone -t 8 -R -r 64K -s 76M -I +Z
Initial write 4354783.88 4392731.31 6337397.50*
Rewrite 4070162.69 3974051.50 7587279.81*
Read 10095324.56 17945227.88* 8359665.56
Re-read 12316555.88 20468303.75* 7949999.34
Reverse Read 4924659.84 8542573.33* 6388858.72
Stride read 10895715.69 14828968.38* 6107484.81
Random read 6838537.34 14352104.25* 5389174.97
Mixed workload 5805646.75 8391745.53* 6052748.25
Random write 4148973.38 3890847.38 7247214.19*
Pwrite 4309372.41 4423800.34 6863604.69*
Pread 4875766.02* 4042375.33 3692948.91
Fwrite 6102404.31 6021884.41 6634112.09*
Fread 15485971.12* 14900780.62 13981842.50
= real 0m45.618s 0m45.753s 0m45.619s
= user 0m0.071s 0m0.080s 0m0.060s
= sys 0m4.702s 0m4.430s 0m3.555s
Test #9 iozone -t 9 -R -r 72K -s 34M -I +Z
Initial write 4202354.67 4208936.34 6300798.88*
Rewrite 4046855.38 4294137.50 7623323.69*
Read 10926571.88 13304801.81* 10895587.19
Re-read 17725984.94* 7964431.25 12394078.50
Reverse Read 5843121.72 5851846.66* 4075657.20
Stride read 9688998.59 10306234.70* 5566376.62
Random read 7656689.97 8660602.06* 5437182.36
Mixed workload 6229215.62 11205238.73* 5575719.75
Random write 4094822.22 4517401.86 6601624.94*
Pwrite 4274497.50 4263936.64 6844453.11*
Pread 6525075.62* 6043725.62 5745003.28
Fwrite 5958798.56 8430354.78* 7636085.00
Fread 18636725.12* 17268959.12 16618803.62
= real 0m44.945s 0m44.816s 0m45.194s
= user 0m0.062s 0m0.060s 0m0.060s
= sys 0m2.187s 0m2.223s 0m1.888s
Test #10 iozone -t 10 -R -r 80K -s 0M -I +Z
Initial write 3213973.56 2731512.62 4416466.25*
Rewrite 3066956.44* 2693819.50 332671.94
Read 7769523.25* 2681473.75 462840.44
Re-read 5244861.75 5473037.00* 382183.03
Reverse Read 7479397.25* 4869597.75 374714.06
Stride read 5403282.50* 5385083.75 382473.44
Random read 5131997.25 5176799.75* 380593.56
Mixed workload 3998043.25 4219049.00* 1645850.45
Random write 3452832.88 3290861.69 3588531.75*
Pwrite 3757435.81 2711756.47 4561807.88*
Pread 2743595.25* 2635835.00 412947.98
Fwrite 16076549.00 16741977.25* 14797209.38
Fread 23581812.62* 21664184.25 5064296.97
= real 0m44.490s 0m44.444s 0m44.609s
= user 0m0.054s 0m0.049s 0m0.055s
= sys 0m0.037s 0m0.046s 0m0.148s
so when the number of active tasks become larger than the number
of online CPUS, iozone reports a bit hard to understand data. I
can assume that since now we keep the preemption disabled longer
in write path, a concurrent operation (READ or WRITE) cannot preempt
current anymore... slightly suspicious.
the other hard to understand thing is why do READ-only tests have
such a huge jitter. READ-only tests don't depend on streams, they
don't even use them, we supply compressed data directly to
decompression api.
may be better retire iozone and never use it again.
"118 insertions(+), 238 deletions(-)" the patches remove a big
pile of code.
-ss
[toc] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-03-25 00:50 +0100 |
| Message-ID | <rgnaO-1wQ-11@gated-at.bofh.it> |
| In reply to | #1363235 |
Hi Sergey,
On Wed, Mar 23, 2016 at 05:18:27PM +0900, Sergey Senozhatsky wrote:
> ( was "[PATCH] zram: export the number of available comp streams"
> forked from http://marc.info/?l=linux-kernel&m=145860707516861 )
>
> d'oh.... sorry, now actually forked.
>
>
> Hello Minchan,
>
> forked into a separate tread.
>
> > On (03/22/16 09:39), Minchan Kim wrote:
> > > zram_bvec_write()
> > > {
> > > *get_cpu_ptr(comp-stream);
> > > zcomp_compress();
> > > zs_malloc()
> > > put_cpu_ptr(comp-stream);
> > > }
> > >
> > > this, however, makes zsmalloc unhapy. pool has GFP_NOIO | __GFP_HIGHMEM
> > > gfp, and GFP_NOIO is ___GFP_DIRECT_RECLAIM|___GFP_KSWAPD_RECLAIM. this
> > > __GFP_DIRECT_RECLAIM is in the conflict with per-cpu streams, because
> > > per-cpu streams require disabled preemption (up until we copy stream
> > > buffer to zspage). so what options do we have here... from the top of
> > > my head (w/o a lot of thinking)...
> >
> > Indeed.
> ...
> > How about this?
> >
> > zram_bvec_write()
> > {
> > retry:
> > *get_cpu_ptr(comp-stream);
> > zcomp_compress();
> > handle = zs_malloc((gfp &~ __GFP_DIRECT_RECLAIM| | GFP_NOWARN)
> > if (!handle) {
> > put_cpu_ptr(comp-stream);
> > handle = zs_malloc(gfp);
> > goto retry;
> > }
> > put_cpu_ptr(comp-stream);
> > }
>
> interesting. the retry jump should go higher, we have "user_mem = kmap_atomic(page)"
> which we unmap right after compression, because a) we don't need
> uncompressed memory anymore b) zs_malloc() can sleep and we can't have atomic
> mapping around. the nasty thing here is is_partial_io(). we need to re-do
>
> if (is_partial_io(bvec))
> memcpy(uncmem + offset, user_mem + bvec-bv_offset,
> bvec-bv_len);
>
> once again in the worst case.
>
> so zs_malloc((gfp &~ __GFP_DIRECT_RECLAIM | GFP_NOWARN) so far can cause
> double memcpy() and double compression. just to outline this.
>
>
> the test.
>
> I executed a number of iozone tests, on each iteration re-creating zram
> device (3GB, LZO, EXT4. the box has 4 x86_64 CPUs).
>
> $DEVICE_SZ=3G
> $FREE_SPACE is 10% of $DEVICE_SZ
> time ./iozone -t $i -R -r $((8*$i))K -s $((($DEVICE_SZ/$i - $FREE_SPACE)/(1024*1024)))M -I +Z
>
>
> columns:
>
> TEST MAX_STREAMS 4 MAX_STREAMS 8 PER_CPU STREAMS
> ====================================================================
>
> Test #1 iozone -t 1 -R -r 8K -s 2764M -I +Z
> Initial write 853492.31* 835868.50 839789.56
> Rewrite 1642073.88 1657255.75 1693011.50*
> Read 3384044.00* 3218727.25 3269109.50
> Re-read 3389794.50* 3243187.00 3267422.25
> Reverse Read 3209805.75* 3082040.00 3107957.25
> Stride read 3100144.50* 2972280.25 2923155.25
> Random read 2992249.75* 2874605.00 2854824.25
> Mixed workload 2992274.75* 2878212.25 2883840.00
> Random write 1471800.00 1452346.50 1515678.75*
> Pwrite 802083.00 801627.31 820251.69*
> Pread 3443495.00* 3308659.25 3302089.00
> Fwrite 1880446.88 1838607.50 1909490.00*
> Fread 3479614.75 3091634.75 6442964.50*
> = real 1m4.170s 1m4.513s 1m4.123s
> = user 0m0.559s 0m0.518s 0m0.511s
> = sys 0m18.766s 0m19.264s 0m18.641s
>
>
> Test #2 iozone -t 2 -R -r 16K -s 1228M -I +Z
> Initial write 2102532.12 2051809.19 2419072.50*
> Rewrite 2217024.25 2250930.00 3681559.00*
> Read 7716933.25 7898759.00 8345507.75*
> Re-read 7748487.75 7765282.25 8342367.50*
> Reverse Read 7415254.25 7552637.25 7822691.75*
> Stride read 7041909.50 7091049.25 7401273.00*
> Random read 6205044.25 6738888.50 7232104.25*
> Mixed workload 4582990.00 5271651.50 5361002.88*
> Random write 2591893.62 2513729.88 3660774.38*
> Pwrite 1873876.75 1909758.69 2087238.81*
> Pread 4669850.00 4651121.56 4919588.44*
> Fwrite 1937947.25 1940628.06 2034251.25*
> Fread 9930319.00 9970078.00* 9831422.50
> = real 0m53.844s 0m53.607s 0m52.528s
> = user 0m0.273s 0m0.289s 0m0.280s
> = sys 0m16.595s 0m16.478s 0m14.072s
>
>
> Test #3 iozone -t 3 -R -r 24K -s 716M -I +Z
> Initial write 3036567.50 2998918.25 3683853.00*
> Rewrite 3402447.88 3415685.88 5054705.38*
> Read 11767413.00* 11133789.50 11246497.25
> Re-read 11797680.50* 11092592.00 11277382.00
> Reverse Read 10828320.00* 10157665.50 10749055.00
> Stride read 10532039.50* 9943521.75 10464700.25
> Random read 10380365.75* 9807859.25 10234127.00
> Mixed workload 8772132.50* 8415083.50 8457108.50
> Random write 3364875.00 3310042.00 5059136.38*
> Pwrite 2677290.25 2651309.50 3198166.25*
> Pread 5221799.56* 4963050.69 4987293.78
> Fwrite 2026887.56 2047679.00 2124199.62*
> Fread 11310381.25 11413531.50 11444208.75*
> = real 0m50.209s 0m50.782s 0m49.750s
> = user 0m0.195s 0m0.205s 0m0.215s
> = sys 0m14.873s 0m15.159s 0m12.911s
>
>
> Test #4 iozone -t 4 -R -r 32K -s 460M -I +Z
> Initial write 3841474.94 3859279.81 5309988.88*
> Rewrite 3905526.25 3917309.62 6814800.62*
> Read 16233054.50 14843560.25 16352283.75*
> Re-read 16335506.50 15529152.25 16352570.00*
> Reverse Read 15316394.50* 14225482.50 15004897.50
> Stride read 14799380.25* 14064034.25 14355184.25
> Random read 14683771.00 14206928.50 14814913.00*
> Mixed workload 9058851.50 9180650.75 10815917.50*
> Random write 3990585.94 4004757.00 6722088.50*
> Pwrite 3318836.12 3468977.69 4244747.69*
> Pread 5894538.16* 5588046.38 5847345.62
> Fwrite 2227353.75 2186688.62 2386974.88*
> Fread 12046094.00 12240004.75* 12073956.75
> = real 0m48.561s 0m48.839s 0m48.142s
> = user 0m0.155s 0m0.170s 0m0.133s
> = sys 0m13.650s 0m13.684s 0m10.790s
>
>
> Test #5 iozone -t 5 -R -r 40K -s 307M -I +Z
> Initial write 4034878.94 4026610.69 5775746.12*
> Rewrite 3898600.44 3901114.16 6923764.19*
> Read 14947360.88 16698824.25* 10155333.62
> Re-read 15844580.75* 15344057.00 9869874.38
> Reverse Read 7459156.95 9023317.86* 7648295.03
> Stride read 10823891.81 9615553.81 11231183.72*
> Random read 10391702.56* 9740935.75 10048038.28
> Mixed workload 8261830.94 10175925.00* 7535763.75
> Random write 3951423.31 3960984.62 6671441.38*
> Pwrite 4119023.12 4097204.56 5975659.12*
> Pread 6072076.73* 4338668.50 6020808.34
> Fwrite 2417235.47 2337875.88 2665450.62*
> Fread 13393630.25 13648332.00* 13395391.00
> = real 0m47.756s 0m47.939s 0m47.483s
> = user 0m0.128s 0m0.128s 0m0.119s
> = sys 0m10.361s 0m10.392s 0m8.717s
>
>
> Test #6 iozone -t 6 -R -r 48K -s 204M -I +Z
> Initial write 4134932.97 4137171.88 5983193.31*
> Rewrite 3928131.31 3950764.00 7124248.00*
> Read 10965005.75* 10152236.50 9856572.88
> Re-read 9386946.00 10776231.38 14303174.12*
> Reverse Read 6035244.89 7456152.38* 5999446.38
> Stride read 8041000.75 7995307.75 10182936.75*
> Random read 8565099.09 10487707.58* 8694877.25
> Mixed workload 5301593.06 7332589.09* 6802251.06
> Random write 4046482.56 3986854.94 6723824.56*
> Pwrite 4188226.41 4214513.34 6245278.44*
> Pread 3452596.86 3708694.69* 3486420.41
> Fwrite 2829500.22 3030742.72 3033792.28*
> Fread 13331387.75 13490416.50 14940410.25*
> = real 0m47.150s 0m47.050s 0m47.044s
> = user 0m0.106s 0m0.100s 0m0.094s
> = sys 0m9.238s 0m8.804s 0m6.930s
>
>
> Test #7 iozone -t 7 -R -r 56K -s 131M -I +Z
> Initial write 4169480.84 4116331.03 5946801.38*
> Rewrite 3993155.97 3986195.00 6928142.44*
> Read 18901600.25* 10088918.69 6699592.78
> Re-read 8738544.69 14881309.62* 13960026.06
> Reverse Read 5008919.08 7923949.95* 5495212.41
> Stride read 7029436.75 8747574.91* 6477087.25
> Random read 6994738.56* 5448687.81 6585235.53
> Mixed workload 5178632.44 5258914.92 5587421.81*
> Random write 4008977.78 3928116.88 6816453.12*
> Pwrite 4342852.09 4154319.09 6124520.06*
> Pread 3880318.99 2978587.56 4493903.14*
> Fwrite 5557990.03 2923556.59 6126649.94*
> Fread 14451722.00 15281179.62* 14675436.50
> = real 0m46.321s 0m46.458s 0m45.791s
> = user 0m0.093s 0m0.089s 0m0.095s
> = sys 0m6.961s 0m6.600s 0m5.499s
>
>
> Test #8 iozone -t 8 -R -r 64K -s 76M -I +Z
> Initial write 4354783.88 4392731.31 6337397.50*
> Rewrite 4070162.69 3974051.50 7587279.81*
> Read 10095324.56 17945227.88* 8359665.56
> Re-read 12316555.88 20468303.75* 7949999.34
> Reverse Read 4924659.84 8542573.33* 6388858.72
> Stride read 10895715.69 14828968.38* 6107484.81
> Random read 6838537.34 14352104.25* 5389174.97
> Mixed workload 5805646.75 8391745.53* 6052748.25
> Random write 4148973.38 3890847.38 7247214.19*
> Pwrite 4309372.41 4423800.34 6863604.69*
> Pread 4875766.02* 4042375.33 3692948.91
> Fwrite 6102404.31 6021884.41 6634112.09*
> Fread 15485971.12* 14900780.62 13981842.50
> = real 0m45.618s 0m45.753s 0m45.619s
> = user 0m0.071s 0m0.080s 0m0.060s
> = sys 0m4.702s 0m4.430s 0m3.555s
>
>
> Test #9 iozone -t 9 -R -r 72K -s 34M -I +Z
> Initial write 4202354.67 4208936.34 6300798.88*
> Rewrite 4046855.38 4294137.50 7623323.69*
> Read 10926571.88 13304801.81* 10895587.19
> Re-read 17725984.94* 7964431.25 12394078.50
> Reverse Read 5843121.72 5851846.66* 4075657.20
> Stride read 9688998.59 10306234.70* 5566376.62
> Random read 7656689.97 8660602.06* 5437182.36
> Mixed workload 6229215.62 11205238.73* 5575719.75
> Random write 4094822.22 4517401.86 6601624.94*
> Pwrite 4274497.50 4263936.64 6844453.11*
> Pread 6525075.62* 6043725.62 5745003.28
> Fwrite 5958798.56 8430354.78* 7636085.00
> Fread 18636725.12* 17268959.12 16618803.62
> = real 0m44.945s 0m44.816s 0m45.194s
> = user 0m0.062s 0m0.060s 0m0.060s
> = sys 0m2.187s 0m2.223s 0m1.888s
>
>
> Test #10 iozone -t 10 -R -r 80K -s 0M -I +Z
> Initial write 3213973.56 2731512.62 4416466.25*
> Rewrite 3066956.44* 2693819.50 332671.94
> Read 7769523.25* 2681473.75 462840.44
> Re-read 5244861.75 5473037.00* 382183.03
> Reverse Read 7479397.25* 4869597.75 374714.06
> Stride read 5403282.50* 5385083.75 382473.44
> Random read 5131997.25 5176799.75* 380593.56
> Mixed workload 3998043.25 4219049.00* 1645850.45
> Random write 3452832.88 3290861.69 3588531.75*
> Pwrite 3757435.81 2711756.47 4561807.88*
> Pread 2743595.25* 2635835.00 412947.98
> Fwrite 16076549.00 16741977.25* 14797209.38
> Fread 23581812.62* 21664184.25 5064296.97
> = real 0m44.490s 0m44.444s 0m44.609s
> = user 0m0.054s 0m0.049s 0m0.055s
> = sys 0m0.037s 0m0.046s 0m0.148s
>
>
> so when the number of active tasks become larger than the number
> of online CPUS, iozone reports a bit hard to understand data. I
> can assume that since now we keep the preemption disabled longer
> in write path, a concurrent operation (READ or WRITE) cannot preempt
> current anymore... slightly suspicious.
>
> the other hard to understand thing is why do READ-only tests have
> such a huge jitter. READ-only tests don't depend on streams, they
> don't even use them, we supply compressed data directly to
> decompression api.
>
> may be better retire iozone and never use it again.
>
>
> "118 insertions(+), 238 deletions(-)" the patches remove a big
> pile of code.
First of all, I appreciate you very much!
At a glance, on write workload, huge win but worth to investigate
how such fluctuation/regression happens on read-related test
(read and mixed workload).
Could you send your patchset? I will test it.
[toc] | [prev] | [next] | [standalone]
| From | Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> |
|---|---|
| Date | 2016-03-25 02:50 +0100 |
| Message-ID | <rgp2V-2Mp-1@gated-at.bofh.it> |
| In reply to | #1364486 |
[Multipart message — attachments visible in raw view] — view raw
Hello Minchan, On (03/25/16 08:41), Minchan Kim wrote: [..] > > Test #10 iozone -t 10 -R -r 80K -s 0M -I +Z > > Initial write 3213973.56 2731512.62 4416466.25* > > Rewrite 3066956.44* 2693819.50 332671.94 > > Read 7769523.25* 2681473.75 462840.44 > > Re-read 5244861.75 5473037.00* 382183.03 > > Reverse Read 7479397.25* 4869597.75 374714.06 > > Stride read 5403282.50* 5385083.75 382473.44 > > Random read 5131997.25 5176799.75* 380593.56 > > Mixed workload 3998043.25 4219049.00* 1645850.45 > > Random write 3452832.88 3290861.69 3588531.75* > > Pwrite 3757435.81 2711756.47 4561807.88* > > Pread 2743595.25* 2635835.00 412947.98 > > Fwrite 16076549.00 16741977.25* 14797209.38 > > Fread 23581812.62* 21664184.25 5064296.97 > > = real 0m44.490s 0m44.444s 0m44.609s > > = user 0m0.054s 0m0.049s 0m0.055s > > = sys 0m0.037s 0m0.046s 0m0.148s > > > > > > so when the number of active tasks become larger than the number > > of online CPUS, iozone reports a bit hard to understand data. I > > can assume that since now we keep the preemption disabled longer > > in write path, a concurrent operation (READ or WRITE) cannot preempt > > current anymore... slightly suspicious. > > > > the other hard to understand thing is why do READ-only tests have > > such a huge jitter. READ-only tests don't depend on streams, they > > don't even use them, we supply compressed data directly to > > decompression api. > > > > may be better retire iozone and never use it again. > > > > > > "118 insertions(+), 238 deletions(-)" the patches remove a big > > pile of code. > > First of all, I appreciate you very much! thanks! > At a glance, on write workload, huge win but worth to investigate > how such fluctuation/regression happens on read-related test > (read and mixed workload). yes, was going to investigate in more details but got interrupted, will return back to it today/tomorrow. > Could you send your patchset? I will test it. oh, sorry, sure! attached (because it's not a real patch submission yet, but they look more or less ready I guess). patches are against next-20160324. -ss
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-03-28 05:30 +0200 |
| Message-ID | <rhw2l-19u-5@gated-at.bofh.it> |
| In reply to | #1364515 |
Hi Sergey,
On Fri, Mar 25, 2016 at 10:47:06AM +0900, Sergey Senozhatsky wrote:
> Hello Minchan,
>
> On (03/25/16 08:41), Minchan Kim wrote:
> [..]
> > > Test #10 iozone -t 10 -R -r 80K -s 0M -I +Z
> > > Initial write 3213973.56 2731512.62 4416466.25*
> > > Rewrite 3066956.44* 2693819.50 332671.94
> > > Read 7769523.25* 2681473.75 462840.44
> > > Re-read 5244861.75 5473037.00* 382183.03
> > > Reverse Read 7479397.25* 4869597.75 374714.06
> > > Stride read 5403282.50* 5385083.75 382473.44
> > > Random read 5131997.25 5176799.75* 380593.56
> > > Mixed workload 3998043.25 4219049.00* 1645850.45
> > > Random write 3452832.88 3290861.69 3588531.75*
> > > Pwrite 3757435.81 2711756.47 4561807.88*
> > > Pread 2743595.25* 2635835.00 412947.98
> > > Fwrite 16076549.00 16741977.25* 14797209.38
> > > Fread 23581812.62* 21664184.25 5064296.97
> > > = real 0m44.490s 0m44.444s 0m44.609s
> > > = user 0m0.054s 0m0.049s 0m0.055s
> > > = sys 0m0.037s 0m0.046s 0m0.148s
> > >
> > >
> > > so when the number of active tasks become larger than the number
> > > of online CPUS, iozone reports a bit hard to understand data. I
> > > can assume that since now we keep the preemption disabled longer
> > > in write path, a concurrent operation (READ or WRITE) cannot preempt
> > > current anymore... slightly suspicious.
> > >
> > > the other hard to understand thing is why do READ-only tests have
> > > such a huge jitter. READ-only tests don't depend on streams, they
> > > don't even use them, we supply compressed data directly to
> > > decompression api.
> > >
> > > may be better retire iozone and never use it again.
> > >
> > >
> > > "118 insertions(+), 238 deletions(-)" the patches remove a big
> > > pile of code.
> >
> > First of all, I appreciate you very much!
>
> thanks!
>
> > At a glance, on write workload, huge win but worth to investigate
> > how such fluctuation/regression happens on read-related test
> > (read and mixed workload).
>
> yes, was going to investigate in more details but got interrupted,
> will return back to it today/tomorrow.
>
> > Could you send your patchset? I will test it.
>
> oh, sorry, sure! attached (because it's not a real patch submission
> yet, but they look more or less ready I guess).
>
> patches are against next-20160324.
Thanks, I tested your patch with fio.
My laptop is 8G ram, 4 CPU.
job file is here.
=
[global]
bs=4k
ioengine=sync
direct=1
size=100m
numjobs=${NUMJOBS}
group_reporting
buffer_compress_percentage=50
filename=/dev/zram0
loops=10
[seq-read]
rw=read
stonewall
[rand-read]
rw=randread
stonewall
[seq-write]
rw=write
stonewall
[rand-write]
rw=randwrite
stonewall
[mixed-seq]
rw=rw
stonewall
[mixed-rand]
rw=randrw
stonewall
=
= old(ie, spinlock) version =
1) NR_PROCESS:8 NR_STREAM: 1
seq-read: (groupid=0, jobs=8): err= 0: pid=23148: Mon Mar 28 12:07:15 2016
read : io=8000.0MB, bw=5925.1MB/s, iops=1517.4K, runt= 1350msec
rand-read: (groupid=1, jobs=8): err= 0: pid=23156: Mon Mar 28 12:07:15 2016
read : io=8000.0MB, bw=4889.1MB/s, iops=1251.9K, runt= 1636msec
seq-write: (groupid=2, jobs=8): err= 0: pid=23164: Mon Mar 28 12:07:15 2016
write: io=8000.0MB, bw=914898KB/s, iops=228724, runt= 8954msec
rand-write: (groupid=3, jobs=8): err= 0: pid=23172: Mon Mar 28 12:07:15 2016
write: io=8000.0MB, bw=913368KB/s, iops=228342, runt= 8969msec
mixed-seq: (groupid=4, jobs=8): err= 0: pid=23180: Mon Mar 28 12:07:15 2016
read : io=4003.1MB, bw=881152KB/s, iops=220287, runt= 4653msec
mixed-rand: (groupid=5, jobs=8): err= 0: pid=23189: Mon Mar 28 12:07:15 2016
read : io=4003.5MB, bw=837491KB/s, iops=209372, runt= 4895msec
2) NR_PROCESS:8 NR_STREAM: 8
seq-read: (groupid=0, jobs=8): err= 0: pid=23248: Mon Mar 28 12:07:57 2016
read : io=8000.0MB, bw=5847.1MB/s, iops=1497.8K, runt= 1368msec
rand-read: (groupid=1, jobs=8): err= 0: pid=23256: Mon Mar 28 12:07:57 2016
read : io=8000.0MB, bw=4778.1MB/s, iops=1223.5K, runt= 1674msec
seq-write: (groupid=2, jobs=8): err= 0: pid=23264: Mon Mar 28 12:07:57 2016
write: io=8000.0MB, bw=1644.7MB/s, iops=420879, runt= 4866msec
rand-write: (groupid=3, jobs=8): err= 0: pid=23272: Mon Mar 28 12:07:57 2016
write: io=8000.0MB, bw=1507.5MB/s, iops=385905, runt= 5307msec
mixed-seq: (groupid=4, jobs=8): err= 0: pid=23280: Mon Mar 28 12:07:57 2016
read : io=4003.1MB, bw=1225.1MB/s, iops=313839, runt= 3266msec
mixed-rand: (groupid=5, jobs=8): err= 0: pid=23288: Mon Mar 28 12:07:57 2016
read : io=4003.5MB, bw=1098.4MB/s, iops=281097, runt= 3646msec
3) NR_PROCESS:8 NR_STREAM: 16
seq-read: (groupid=0, jobs=8): err= 0: pid=23350: Mon Mar 28 12:08:38 2016
read : io=8000.0MB, bw=5843.7MB/s, iops=1495.1K, runt= 1369msec
rand-read: (groupid=1, jobs=8): err= 0: pid=23358: Mon Mar 28 12:08:38 2016
read : io=8000.0MB, bw=4810.6MB/s, iops=1231.6K, runt= 1663msec
seq-write: (groupid=2, jobs=8): err= 0: pid=23366: Mon Mar 28 12:08:38 2016
write: io=8000.0MB, bw=1655.7MB/s, iops=423841, runt= 4832msec
rand-write: (groupid=3, jobs=8): err= 0: pid=23374: Mon Mar 28 12:08:38 2016
write: io=8000.0MB, bw=1501.6MB/s, iops=384384, runt= 5328msec
mixed-seq: (groupid=4, jobs=8): err= 0: pid=23382: Mon Mar 28 12:08:38 2016
read : io=4003.1MB, bw=1221.9MB/s, iops=312786, runt= 3277msec
mixed-rand: (groupid=5, jobs=8): err= 0: pid=23390: Mon Mar 28 12:08:38 2016
read : io=4003.5MB, bw=1104.1MB/s, iops=282647, runt= 3626msec
= percpu =
1) NR_PROCESS:8
seq-read: (groupid=0, jobs=8): err= 0: pid=22804: Mon Mar 28 11:58:22 2016
read : io=8000.0MB, bw=5610.1MB/s, iops=1436.2K, runt= 1426msec
rand-read: (groupid=1, jobs=8): err= 0: pid=22812: Mon Mar 28 11:58:22 2016
read : io=8000.0MB, bw=4881.3MB/s, iops=1249.6K, runt= 1639msec
seq-write: (groupid=2, jobs=8): err= 0: pid=22820: Mon Mar 28 11:58:22 2016
write: io=8000.0MB, bw=1814.6MB/s, iops=464399, runt= 4410msec
rand-write: (groupid=3, jobs=8): err= 0: pid=22829: Mon Mar 28 11:58:22 2016
write: io=8000.0MB, bw=1647.9MB/s, iops=421833, runt= 4855msec
mixed-seq: (groupid=4, jobs=8): err= 0: pid=22837: Mon Mar 28 11:58:22 2016
read : io=4003.1MB, bw=1275.2MB/s, iops=326433, runt= 3140msec
mixed-rand: (groupid=5, jobs=8): err= 0: pid=22846: Mon Mar 28 11:58:22 2016
read : io=4003.5MB, bw=1119.3MB/s, iops=286519, runt= 3577msec
In my test, read is stable. It seems iozone or fs made noise in your test.
Benefit cause by per-cpu on write side is about 10% which is not huge
compared to your previous post.
Hmm, Could you retest to who how the benefit is big?
[toc] | [prev] | [next] | [standalone]
| From | Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> |
|---|---|
| Date | 2016-03-30 10:40 +0200 |
| Message-ID | <rijPu-2H7-35@gated-at.bofh.it> |
| In reply to | #1365288 |
Hello Minchan,
sorry for long reply.
On (03/28/16 12:21), Minchan Kim wrote:
[..]
> group_reporting
> buffer_compress_percentage=50
> filename=/dev/zram0
> loops=10
I used a bit different script. no `buffer_compress_percentage' option,
because it provide "a mix of random data and zeroes"
buffer_compress_percentage=int
If this is set, then fio will attempt to provide IO buffer content
(on WRITEs) that compress to the specified level. Fio does this by
providing a mix of random data and zeroes
and I also used scramble_buffers=0. but default scramble_buffers is
true, so
scramble_buffers=bool
If refill_buffers is too costly and the target is using data
deduplication, then setting this option will slightly modify the IO
buffer contents to defeat normal de-dupe attempts. This is not
enough to defeat more clever block compression attempts, but it will
stop naive dedupe of blocks. Default: true.
hm, but I guess it's not enough; fio probably will have different
data (well, only if we didn't ask it to zero-fill the buffers) for
different tests, causing different zram->zsmalloc behaviour. need
to check it.
> Hmm, Could you retest to who how the benefit is big?
sure. the results are:
- seq-read
- rand-read
- seq-write
- rand-write (READ + WRITE)
- mixed-seq
- mixed-rand (READ + WRITE)
TEST 4 streams 8 streams per-cpu
#jobs1
READ: 2665.4MB/s 2515.2MB/s 2632.4MB/s
READ: 2258.2MB/s 2055.2MB/s 2166.2MB/s
WRITE: 933180KB/s 894260KB/s 898234KB/s
WRITE: 765576KB/s 728154KB/s 746396KB/s
READ: 563169KB/s 541004KB/s 551541KB/s
WRITE: 562660KB/s 540515KB/s 551043KB/s
READ: 493656KB/s 477990KB/s 488041KB/s
WRITE: 493210KB/s 477558KB/s 487600KB/s
#jobs2
READ: 5116.7MB/s 4607.1MB/s 4401.5MB/s
READ: 4401.5MB/s 3993.6MB/s 3831.6MB/s
WRITE: 1539.9MB/s 1425.5MB/s 1600.0MB/s
WRITE: 1311.1MB/s 1228.7MB/s 1380.6MB/s
READ: 1001.8MB/s 960799KB/s 989.63MB/s
WRITE: 998.31MB/s 957540KB/s 986.26MB/s
READ: 921439KB/s 860387KB/s 899720KB/s
WRITE: 918314KB/s 857469KB/s 896668KB/s
#jobs3
READ: 6670.9MB/s 6469.9MB/s 6548.8MB/s
READ: 5743.4MB/s 5507.8MB/s 5608.4MB/s
WRITE: 1923.8MB/s 1885.9MB/s 2191.9MB/s
WRITE: 1622.4MB/s 1605.4MB/s 1842.2MB/s
READ: 1277.3MB/s 1295.8MB/s 1395.2MB/s
WRITE: 1276.9MB/s 1295.4MB/s 1394.7MB/s
READ: 1152.6MB/s 1137.1MB/s 1216.6MB/s
WRITE: 1152.2MB/s 1137.6MB/s 1216.2MB/s
#jobs4
READ: 8720.4MB/s 7301.7MB/s 7896.2MB/s
READ: 7510.3MB/s 6690.1MB/s 6456.2MB/s
WRITE: 2211.6MB/s 1930.8MB/s 2713.9MB/s
WRITE: 2002.2MB/s 1629.8MB/s 2227.7MB/s
READ: 1657.8MB/s 1437.1MB/s 1765.8MB/s
WRITE: 1651.7MB/s 1432.7MB/s 1759.3MB/s
READ: 1467.7MB/s 1201.7MB/s 1523.5MB/s
WRITE: 1462.3MB/s 1197.3MB/s 1517.9MB/s
#jobs5
READ: 7791.9MB/s 6852.7MB/s 7487.9MB/s
READ: 6214.6MB/s 6449.6MB/s 7106.5MB/s
WRITE: 2017.9MB/s 1978.1MB/s 2221.5MB/s
WRITE: 1913.1MB/s 1664.9MB/s 1985.8MB/s
READ: 1417.6MB/s 1447.7MB/s 1558.8MB/s
WRITE: 1419.8MB/s 1449.3MB/s 1561.2MB/s
READ: 1336.9MB/s 1234.1MB/s 1404.7MB/s
WRITE: 1338.2MB/s 1236.9MB/s 1406.8MB/s
#jobs6
READ: 8680.9MB/s 8500.0MB/s 7116.3MB/s
READ: 7329.4MB/s 6580.7MB/s 6476.2MB/s
WRITE: 2121.4MB/s 1918.6MB/s 2472.8MB/s
WRITE: 1936.8MB/s 1826.9MB/s 2106.8MB/s
READ: 1559.9MB/s 1506.3MB/s 1643.6MB/s
WRITE: 1554.7MB/s 1501.2MB/s 1637.2MB/s
READ: 1459.7MB/s 1258.9MB/s 1502.6MB/s
WRITE: 1454.8MB/s 1254.6MB/s 1497.5MB/s
#jobs7
READ: 9170.0MB/s 7905.2MB/s 8043.9MB/s
READ: 6412.7MB/s 6792.7MB/s 6457.8MB/s
WRITE: 2042.4MB/s 1972.5MB/s 2400.6MB/s
WRITE: 1938.8MB/s 1808.7MB/s 2152.6MB/s
READ: 1634.9MB/s 1505.8MB/s 1746.4MB/s
WRITE: 1640.1MB/s 1511.4MB/s 1753.7MB/s
READ: 1407.9MB/s 1239.1MB/s 1480.8MB/s
WRITE: 1413.8MB/s 1245.2MB/s 1486.1MB/s
#jobs8
READ: 8563.4MB/s 8106.7MB/s 7696.3MB/s
READ: 6909.1MB/s 5790.5MB/s 6537.7MB/s
WRITE: 2040.3MB/s 2061.2MB/s 2481.7MB/s
WRITE: 1993.5MB/s 1859.4MB/s 2171.5MB/s
READ: 1691.6MB/s 1585.8MB/s 1749.9MB/s
WRITE: 1686.3MB/s 1580.1MB/s 1744.5MB/s
READ: 1478.3MB/s 1365.4MB/s 1529.2MB/s
WRITE: 1473.5MB/s 1361.2MB/s 1525.3MB/s
#jobs9
READ: 9272.8MB/s 11769MB/s 8052.7MB/s
READ: 6954.6MB/s 6375.0MB/s 7116.3MB/s
WRITE: 2170.3MB/s 1961.6MB/s 2487.9MB/s
WRITE: 1865.9MB/s 1974.2MB/s 2233.6MB/s
READ: 1663.9MB/s 1561.1MB/s 1719.1MB/s
WRITE: 1662.3MB/s 1560.6MB/s 1718.4MB/s
READ: 1515.6MB/s 1379.2MB/s 1546.2MB/s
WRITE: 1514.2MB/s 1377.8MB/s 1544.8MB/s
#jobs10
READ: 8737.3MB/s 7950.4MB/s 7915.5MB/s
READ: 6884.9MB/s 6332.8MB/s 6334.3MB/s
WRITE: 2077.9MB/s 2005.4MB/s 2509.5MB/s
WRITE: 1893.3MB/s 1935.9MB/s 2155.4MB/s
READ: 1519.2MB/s 1548.1MB/s 1622.9MB/s
WRITE: 1519.5MB/s 1549.2MB/s 1623.2MB/s
READ: 1383.4MB/s 1385.2MB/s 1437.6MB/s
WRITE: 1383.6MB/s 1385.4MB/s 1437.8MB/s
-ss
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-03-31 00:20 +0200 |
| Message-ID | <riwCZ-3sy-3@gated-at.bofh.it> |
| In reply to | #1367006 |
On Wed, Mar 30, 2016 at 05:34:19PM +0900, Sergey Senozhatsky wrote: > Hello Minchan, > sorry for long reply. > > On (03/28/16 12:21), Minchan Kim wrote: > [..] > > group_reporting > > buffer_compress_percentage=50 > > filename=/dev/zram0 > > loops=10 > > I used a bit different script. no `buffer_compress_percentage' option, > because it provide "a mix of random data and zeroes" Normally, zram's compression ratio is 3 or 2 so I used it. Hmm, isn't it more real practice usecase? If we don't use buffer_compress_percentage, what's the content in the buffer? > > buffer_compress_percentage=int > If this is set, then fio will attempt to provide IO buffer content > (on WRITEs) that compress to the specified level. Fio does this by > providing a mix of random data and zeroes > > and I also used scramble_buffers=0. but default scramble_buffers is > true, so > > scramble_buffers=bool > If refill_buffers is too costly and the target is using data > deduplication, then setting this option will slightly modify the IO > buffer contents to defeat normal de-dupe attempts. This is not > enough to defeat more clever block compression attempts, but it will > stop naive dedupe of blocks. Default: true. > > hm, but I guess it's not enough; fio probably will have different > data (well, only if we didn't ask it to zero-fill the buffers) for > different tests, causing different zram->zsmalloc behaviour. need > to check it. > > > > Hmm, Could you retest to who how the benefit is big? > > sure. the results are: > > - seq-read > - rand-read > - seq-write > - rand-write (READ + WRITE) > - mixed-seq > - mixed-rand (READ + WRITE) > > TEST 4 streams 8 streams per-cpu > > #jobs1 > READ: 2665.4MB/s 2515.2MB/s 2632.4MB/s > READ: 2258.2MB/s 2055.2MB/s 2166.2MB/s > WRITE: 933180KB/s 894260KB/s 898234KB/s > WRITE: 765576KB/s 728154KB/s 746396KB/s > READ: 563169KB/s 541004KB/s 551541KB/s > WRITE: 562660KB/s 540515KB/s 551043KB/s > READ: 493656KB/s 477990KB/s 488041KB/s > WRITE: 493210KB/s 477558KB/s 487600KB/s > #jobs2 > READ: 5116.7MB/s 4607.1MB/s 4401.5MB/s > READ: 4401.5MB/s 3993.6MB/s 3831.6MB/s > WRITE: 1539.9MB/s 1425.5MB/s 1600.0MB/s > WRITE: 1311.1MB/s 1228.7MB/s 1380.6MB/s > READ: 1001.8MB/s 960799KB/s 989.63MB/s > WRITE: 998.31MB/s 957540KB/s 986.26MB/s > READ: 921439KB/s 860387KB/s 899720KB/s > WRITE: 918314KB/s 857469KB/s 896668KB/s > #jobs3 > READ: 6670.9MB/s 6469.9MB/s 6548.8MB/s > READ: 5743.4MB/s 5507.8MB/s 5608.4MB/s > WRITE: 1923.8MB/s 1885.9MB/s 2191.9MB/s > WRITE: 1622.4MB/s 1605.4MB/s 1842.2MB/s > READ: 1277.3MB/s 1295.8MB/s 1395.2MB/s > WRITE: 1276.9MB/s 1295.4MB/s 1394.7MB/s > READ: 1152.6MB/s 1137.1MB/s 1216.6MB/s > WRITE: 1152.2MB/s 1137.6MB/s 1216.2MB/s > #jobs4 > READ: 8720.4MB/s 7301.7MB/s 7896.2MB/s > READ: 7510.3MB/s 6690.1MB/s 6456.2MB/s > WRITE: 2211.6MB/s 1930.8MB/s 2713.9MB/s > WRITE: 2002.2MB/s 1629.8MB/s 2227.7MB/s Your case is 40% win. It's huge, Nice! I tested with your guide line(i.e., no buffer_compress_percentage, scramble_buffers=0) but still 10% enhance in my machine. Hmm,,, How about if you test my fio job.file in your machine? Still, it's 40% win? Also, I want to test again in your exactly same configuration. Could you tell me zram environment(ie, disksize, compression algorithm) and share me your job.file of fio? Thanks.
[toc] | [prev] | [next] | [standalone]
| From | Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> |
|---|---|
| Date | 2016-03-31 03:30 +0200 |
| Message-ID | <rizAT-5C4-15@gated-at.bofh.it> |
| In reply to | #1367636 |
Hello,
On (03/31/16 07:12), Minchan Kim wrote:
[..]
> > I used a bit different script. no `buffer_compress_percentage' option,
> > because it provide "a mix of random data and zeroes"
>
> Normally, zram's compression ratio is 3 or 2 so I used it.
> Hmm, isn't it more real practice usecase?
this option guarantees that the supplied to zram data will have
a requested compression ratio? hm, but we never do that in real
life, zram sees random data.
> If we don't use buffer_compress_percentage, what's the content in the buffer?
that's a good question. I quickly looked into the fio source code,
we need to use "buffer_pattern=str" option, I think. so the buffers
will be filled with the same data.
I don't mind to have buffer_compress_percentage as a separate test (set
as a local test option), but I think that using common buffer pattern
adds more confidence when we compare test results.
[..]
> > hm, but I guess it's not enough; fio probably will have different
> > data (well, only if we didn't ask it to zero-fill the buffers) for
> > different tests, causing different zram->zsmalloc behaviour. need
> > to check it.
[..]
> > #jobs4
> > READ: 8720.4MB/s 7301.7MB/s 7896.2MB/s
> > READ: 7510.3MB/s 6690.1MB/s 6456.2MB/s
> > WRITE: 2211.6MB/s 1930.8MB/s 2713.9MB/s
> > WRITE: 2002.2MB/s 1629.8MB/s 2227.7MB/s
>
> Your case is 40% win. It's huge, Nice!
> I tested with your guide line(i.e., no buffer_compress_percentage,
> scramble_buffers=0) but still 10% enhance in my machine.
> Hmm,,,
>
> How about if you test my fio job.file in your machine?
> Still, it's 40% win?
I'll retest with new config.
> Also, I want to test again in your exactly same configuration.
> Could you tell me zram environment(ie, disksize, compression
> algorithm) and share me your job.file of fio?
sure.
3G, lzo
--- my fio-template is
[global]
bs=4k
ioengine=sync
direct=1
size=__SIZE__
numjobs=__JOBS__
group_reporting
filename=/dev/zram0
loops=1
buffer_pattern=0xbadc0ffee
scramble_buffers=0
[seq-read]
rw=read
stonewall
[rand-read]
rw=randread
stonewall
[seq-write]
rw=write
stonewall
[rand-write]
rw=randwrite
stonewall
[mixed-seq]
rw=rw
stonewall
[mixed-rand]
rw=randrw
stonewall
#separate test with
#buffer_compress_percentage=50
--- my create-zram script is as follows.
#!/bin/sh
rmmod zram
modprobe zram
if [ -e /sys/block/zram0/initstate ]; then
initdone=`cat /sys/block/zram0/initstate`
if [ $initdone = 1 ]; then
echo "init done"
exit 1
fi
fi
echo 8 > /sys/block/zram0/max_comp_streams
echo lzo > /sys/block/zram0/comp_algorithm
cat /sys/block/zram0/comp_algorithm
cat /sys/block/zram0/max_comp_streams
echo $1 > /sys/block/zram0/disksize
--- and I use it as
#!/bin/sh
DEVICE_SZ=$((3 * 1024 * 1024 * 1024))
FREE_SPACE=$(($DEVICE_SZ / 10))
LOG=/tmp/fio-zram-test
LOG_SUFFIX=$1
function reset_zram
{
rmmod zram
}
function create_zram
{
./create-zram $DEVICE_SZ
}
function main
{
local j
local i
if [ "z$LOG_SUFFIX" = "z" ]; then
LOG_SUFFIX="UNSET"
fi
LOG=$LOG-$LOG_SUFFIX
for i in {1..10}; do
reset_zram
create_zram
cat fio-test-template | sed s/__JOBS__/$i/ | sed s/__SIZE__/$((($DEVICE_SZ/$i - $FREE_SPACE)/(1024*1024)))M/ > fio-test
echo "#jobs$i" >> $LOG
time fio ./fio-test >> $LOG
done
reset_zram
}
main
-- then I use this simple script
#!/bin/sh
if [ "z$2" = "z" ]; then
cat $1 | egrep "#jobs|READ|WRITE" | awk '{printf "%-15s %15s\n", $1, $3}' | sed s/aggrb=// | sed s/,//
else
cat $1 | egrep "#jobs|READ|WRITE" | awk '{printf " %-15s\n", $3}' | sed s/aggrb=// | sed s/\#jobs[0-9]*// | sed s/,//
fi
as
./squeeze.sh fio-zram-test-4-stream > 4s
./squeeze.sh fio-zram-test-8-stream A > 8s
./squeeze.sh fio-zram-test-per-cpu A > pc
and
paste 4s 8s pc > result
-ss
[toc] | [prev] | [next] | [standalone]
| From | Minchan Kim <minchan@kernel.org> |
|---|---|
| Date | 2016-03-31 08:00 +0200 |
| Message-ID | <riDOb-iF-21@gated-at.bofh.it> |
| In reply to | #1367738 |
Hello Sergey,
On Thu, Mar 31, 2016 at 10:26:26AM +0900, Sergey Senozhatsky wrote:
> Hello,
>
> On (03/31/16 07:12), Minchan Kim wrote:
> [..]
> > > I used a bit different script. no `buffer_compress_percentage' option,
> > > because it provide "a mix of random data and zeroes"
> >
> > Normally, zram's compression ratio is 3 or 2 so I used it.
> > Hmm, isn't it more real practice usecase?
>
> this option guarantees that the supplied to zram data will have
> a requested compression ratio? hm, but we never do that in real
> life, zram sees random data.
I agree it's hard to create such random read data with benchmark.
One option is that we share swap dump data of real product, for exmaple,
android or webOS and feed it to the benchmark. But as you know, it
cannot cover all of workload, either. So, to just easy test, I wanted
to make represntative compression ratio data and fio provides option
for it via buffer_compress_percentage.
It would be better rather than feeding random data which could make
lots of noise for each test cycle.
>
> > If we don't use buffer_compress_percentage, what's the content in the buffer?
>
> that's a good question. I quickly looked into the fio source code,
> we need to use "buffer_pattern=str" option, I think. so the buffers
> will be filled with the same data.
>
> I don't mind to have buffer_compress_percentage as a separate test (set
> as a local test option), but I think that using common buffer pattern
> adds more confidence when we compare test results.
If we both uses same "buffer_compress_percentage=something", it's
good to compare. The benefit of buffer_compress_percentage is we can
change compression ratio easily in zram testing and see various
test to see what compression ratio or speed affects the system.
>
> [..]
> > > hm, but I guess it's not enough; fio probably will have different
> > > data (well, only if we didn't ask it to zero-fill the buffers) for
> > > different tests, causing different zram->zsmalloc behaviour. need
> > > to check it.
> [..]
> > > #jobs4
> > > READ: 8720.4MB/s 7301.7MB/s 7896.2MB/s
> > > READ: 7510.3MB/s 6690.1MB/s 6456.2MB/s
> > > WRITE: 2211.6MB/s 1930.8MB/s 2713.9MB/s
> > > WRITE: 2002.2MB/s 1629.8MB/s 2227.7MB/s
> >
> > Your case is 40% win. It's huge, Nice!
> > I tested with your guide line(i.e., no buffer_compress_percentage,
> > scramble_buffers=0) but still 10% enhance in my machine.
> > Hmm,,,
> >
> > How about if you test my fio job.file in your machine?
> > Still, it's 40% win?
>
> I'll retest with new config.
>
> > Also, I want to test again in your exactly same configuration.
> > Could you tell me zram environment(ie, disksize, compression
> > algorithm) and share me your job.file of fio?
>
> sure.
I tested with you suggested parameter.
In my side, win is better compared to my previous test but it seems
your test is so fast. IOW, filesize is small and loops is just 1.
Please test filesize=500m loops=10 or 20.
It can make your test more stable and enhance is 10~20% in my side.
Let's discuss further once test result between us is consistent.
Thanks.
>
> 3G, lzo
>
>
> --- my fio-template is
>
> [global]
> bs=4k
> ioengine=sync
> direct=1
> size=__SIZE__
> numjobs=__JOBS__
> group_reporting
> filename=/dev/zram0
> loops=1
> buffer_pattern=0xbadc0ffee
> scramble_buffers=0
>
> [seq-read]
> rw=read
> stonewall
>
> [rand-read]
> rw=randread
> stonewall
>
> [seq-write]
> rw=write
> stonewall
>
> [rand-write]
> rw=randwrite
> stonewall
>
> [mixed-seq]
> rw=rw
> stonewall
>
> [mixed-rand]
> rw=randrw
> stonewall
>
>
> #separate test with
> #buffer_compress_percentage=50
>
>
>
> --- my create-zram script is as follows.
>
>
> #!/bin/sh
>
> rmmod zram
> modprobe zram
>
> if [ -e /sys/block/zram0/initstate ]; then
> initdone=`cat /sys/block/zram0/initstate`
> if [ $initdone = 1 ]; then
> echo "init done"
> exit 1
> fi
> fi
>
> echo 8 > /sys/block/zram0/max_comp_streams
>
> echo lzo > /sys/block/zram0/comp_algorithm
> cat /sys/block/zram0/comp_algorithm
>
> cat /sys/block/zram0/max_comp_streams
> echo $1 > /sys/block/zram0/disksize
>
>
>
>
>
> --- and I use it as
>
>
> #!/bin/sh
>
> DEVICE_SZ=$((3 * 1024 * 1024 * 1024))
> FREE_SPACE=$(($DEVICE_SZ / 10))
> LOG=/tmp/fio-zram-test
> LOG_SUFFIX=$1
>
> function reset_zram
> {
> rmmod zram
> }
>
> function create_zram
> {
> ./create-zram $DEVICE_SZ
> }
>
> function main
> {
> local j
> local i
>
> if [ "z$LOG_SUFFIX" = "z" ]; then
> LOG_SUFFIX="UNSET"
> fi
>
> LOG=$LOG-$LOG_SUFFIX
>
> for i in {1..10}; do
> reset_zram
> create_zram
>
> cat fio-test-template | sed s/__JOBS__/$i/ | sed s/__SIZE__/$((($DEVICE_SZ/$i - $FREE_SPACE)/(1024*1024)))M/ > fio-test
> echo "#jobs$i" >> $LOG
> time fio ./fio-test >> $LOG
> done
>
> reset_zram
> }
>
> main
>
>
>
>
> -- then I use this simple script
>
> #!/bin/sh
>
> if [ "z$2" = "z" ]; then
> cat $1 | egrep "#jobs|READ|WRITE" | awk '{printf "%-15s %15s\n", $1, $3}' | sed s/aggrb=// | sed s/,//
> else
> cat $1 | egrep "#jobs|READ|WRITE" | awk '{printf " %-15s\n", $3}' | sed s/aggrb=// | sed s/\#jobs[0-9]*// | sed s/,//
> fi
>
>
>
>
> as
>
> ./squeeze.sh fio-zram-test-4-stream > 4s
> ./squeeze.sh fio-zram-test-8-stream A > 8s
> ./squeeze.sh fio-zram-test-per-cpu A > pc
>
> and
>
> paste 4s 8s pc > result
>
>
> -ss
[toc] | [prev] | [next] | [standalone]
| From | Sergey Senozhatsky <sergey.senozhatsky.work@gmail.com> |
|---|---|
| Date | 2016-03-31 08:40 +0200 |
| Message-ID | <riEqT-P8-39@gated-at.bofh.it> |
| In reply to | #1367829 |
Hello Minchan, On (03/31/16 14:53), Minchan Kim wrote: > Hello Sergey, > > > that's a good question. I quickly looked into the fio source code, > > we need to use "buffer_pattern=str" option, I think. so the buffers > > will be filled with the same data. > > > > I don't mind to have buffer_compress_percentage as a separate test (set > > as a local test option), but I think that using common buffer pattern > > adds more confidence when we compare test results. > > If we both uses same "buffer_compress_percentage=something", it's > good to compare. The benefit of buffer_compress_percentage is we can > change compression ratio easily in zram testing and see various > test to see what compression ratio or speed affects the system. let's start with "common data" (buffer_pattern=str), not common compression ratio. buffer_compress_percentage=something is calculated for which compression algorithm? deflate (zlib)? or it's something else? we use lzo/lz4, common data is more predictable. [..] > > sure. > > I tested with you suggested parameter. > In my side, win is better compared to my previous test but it seems > your test is so fast. IOW, filesize is small and loops is just 1. > Please test filesize=500m loops=10 or 20. that will require 5G zram, I don't have that much ram on the box so I'll test later today on another box. I split the device size between jobs. if I have 10 jobs, then the file size of each job is DISK_SIZE/10; but in total jobs write/read DEVICE_SZ bytes. jobs start with large 1 * DEVICE_SZ/1 files and go down to 10 * DEVICE_SZ/10 files. > It can make your test more stable and enhance is 10~20% in my side. > Let's discuss further once test result between us is consistent. -ss
[toc] | [prev] | [next] | [standalone]
| From | Sergey Senozhatsky <sergey.senozhatsky@gmail.com> |
|---|---|
| Date | 2016-04-01 17:50 +0200 |
| Message-ID | <rj9uF-6rC-9@gated-at.bofh.it> |
| In reply to | #1367861 |
Hello Minchan, On (03/31/16 15:34), Sergey Senozhatsky wrote: > > I tested with you suggested parameter. > > In my side, win is better compared to my previous test but it seems > > your test is so fast. IOW, filesize is small and loops is just 1. > > Please test filesize=500m loops=10 or 20. fio - loops=10 - buffer_pattern=0xbadc0ffee zram 6G. no intel p-state, deadline IO scheduler, no lockdep (no lock debugging). test 8 streams per-cpu #jobs1 READ: 4118.2MB/s 4105.3MB/s READ: 3487.7MB/s 3624.9MB/s WRITE: 2197.8MB/s 2305.1MB/s WRITE: 1776.2MB/s 1887.5MB/s READ: 736589KB/s 745648KB/s WRITE: 736353KB/s 745409KB/s READ: 679279KB/s 686559KB/s WRITE: 679093KB/s 686371KB/s #jobs2 READ: 6924.6MB/s 7160.2MB/s READ: 6213.2MB/s 6247.1MB/s WRITE: 2510.3MB/s 3680.1MB/s WRITE: 2286.2MB/s 3153.9MB/s READ: 1163.1MB/s 1333.7MB/s WRITE: 1163.4MB/s 1332.2MB/s READ: 1122.9MB/s 1240.3MB/s WRITE: 1121.9MB/s 1239.2MB/s #jobs3 READ: 10304MB/s 10424MB/s READ: 9014.5MB/s 9014.5MB/s WRITE: 3883.9MB/s 5373.8MB/s WRITE: 3549.1MB/s 4576.4MB/s READ: 1704.4MB/s 1916.8MB/s WRITE: 1704.9MB/s 1915.9MB/s READ: 1603.5MB/s 1806.8MB/s WRITE: 1598.8MB/s 1800.8MB/s #jobs4 READ: 13509MB/s 12792MB/s READ: 10899MB/s 11434MB/s WRITE: 4027.2MB/s 6272.8MB/s WRITE: 3902.1MB/s 5389.2MB/s READ: 2090.9MB/s 2344.4MB/s WRITE: 2085.2MB/s 2337.1MB/s READ: 1968.1MB/s 2185.9MB/s WRITE: 1969.5MB/s 2186.4MB/s #jobs5 READ: 12634MB/s 11607MB/s READ: 9932.7MB/s 9980.6MB/s WRITE: 4275.8MB/s 5844.3MB/s WRITE: 4210.1MB/s 5262.3MB/s READ: 1995.6MB/s 2211.4MB/s WRITE: 1988.4MB/s 2203.4MB/s READ: 1930.1MB/s 2191.8MB/s WRITE: 1929.8MB/s 2190.3MB/s #jobs6 READ: 12270MB/s 13012MB/s READ: 11221MB/s 10815MB/s WRITE: 4643.4MB/s 6090.9MB/s WRITE: 4373.6MB/s 5772.8MB/s READ: 2232.6MB/s 2358.4MB/s WRITE: 2233.4MB/s 2359.2MB/s READ: 2082.6MB/s 2285.8MB/s WRITE: 2075.9MB/s 2278.1MB/s #jobs7 READ: 13617MB/s 14172MB/s READ: 12290MB/s 11734MB/s WRITE: 5077.3MB/s 6315.7MB/s WRITE: 4719.4MB/s 5825.1MB/s READ: 2379.8MB/s 2523.7MB/s WRITE: 2373.7MB/s 2516.7MB/s READ: 2287.9MB/s 2362.4MB/s WRITE: 2283.9MB/s 2358.2MB/s #jobs8 READ: 15130MB/s 15533MB/s READ: 12952MB/s 13077MB/s WRITE: 5586.6MB/s 7108.2MB/s WRITE: 5233.5MB/s 6591.3MB/s READ: 2541.2MB/s 2709.2MB/s WRITE: 2544.6MB/s 2713.2MB/s READ: 2450.6MB/s 2590.7MB/s WRITE: 2449.4MB/s 2589.3MB/s #jobs9 READ: 13480MB/s 13909MB/s READ: 12389MB/s 12000MB/s WRITE: 5266.8MB/s 6594.9MB/s WRITE: 4971.6MB/s 6442.2MB/s READ: 2464.9MB/s 2470.9MB/s WRITE: 2482.7MB/s 2488.8MB/s READ: 2171.9MB/s 2402.2MB/s WRITE: 2174.9MB/s 2405.5MB/s #jobs10 READ: 14647MB/s 14667MB/s READ: 11765MB/s 12032MB/s WRITE: 5248.7MB/s 6740.4MB/s WRITE: 4779.8MB/s 5822.8MB/s READ: 2448.8MB/s 2585.3MB/s WRITE: 2449.4MB/s 2585.9MB/s READ: 2290.5MB/s 2409.1MB/s WRITE: 2290.2MB/s 2409.7MB/s -ss
[toc] | [prev] | [standalone]
Back to top | Article view | linux.kernel
csiph-web