Re: [sot] file times / file metadata

From	cross@spitfire.i.gajendra.net (Dan Cross)
Newsgroups	comp.lang.c
Subject	Re: [sot] file times / file metadata
Date	2017-11-30 17:26 +0000
Organization	PANIX Public Access Internet and UNIX, NYC
Message-ID	<ovpevv$aud$1@reader2.panix.com> (permalink)
References	<5a91718b-b272-4cdf-adfd-8f6777851b23@googlegroups.com> <ovlthi$ptt$1@dont-email.me> <ovn77c$41o$1@reader2.panix.com> <ovnbf7$ecf$1@dont-email.me>

Show all headers | View raw

In article <ovnbf7$ecf$1@dont-email.me>,
David Brown  <david.brown@hesbynett.no> wrote:
>On 29/11/17 22:01, Dan Cross wrote:
>> Indeed. However, the file server was a dedicated machine on the
>> network, and it was found that it could still saturate the network
>> interface. Furthermore, there was a write-cache placed in front of
>> Venti to amortize the cost of transient storage.
>
>If the storage is spinning rust, then with relatively modern processors 
>and enough ram, it is worth making considerable effort to reduce the 
>disk transfers.  Transparent compression (such as btrfs and zfs support) 
>means more cpu work, but can make reads and writes significantly faster 
>due to smaller disk transfers.  I expect the same will apply to this 
>dedup'ed server if there is a lot of similarity in the files.  But it 
>will depend on the relative speeds of the components in the server.

The venti paper is now 15 years old (presented at FAST in 2002;
paper itself here: http://9p.io/sys/doc/venti/venti.pdf).  CPUs
were slower and RAM was smaller back then.

At the time, the transition to the new filesystem and Venti was
something of a radical departure: the Plan 9 file server had
always been a dedicated machine, but previously it had run a
specialized kernel that did not permit arbitrary user processes.
Moving to venti meant that a "normal" CPU server kernel could be
used for file service; that was pretty cool but concerns about
increased latency were raised: in the previous file server,
basically all of physical memory was allocated to a filesystem
cache, all of mag disk was a cache, and the actual filesystem
lived on a WORM device (at Bell Labs and a few other sites, this
was an actual magneto-optical WORM robot; most places used a
'pseudo-cache' backed by magdisk).  In the new scheme, there
would be scheduler interference and potentially other processes,
though by this time even for modest sites it was often cheap
enough to dedicate a machine to file service: the network was
the bottleneck and as long as the software could keep the
network device saturated, little concern was given to optimizing
IO to/from the disk devices.

That said, balancing IO between the index and data "arenas" was
recognized pretty early on as a bottleneck, and much attention
and care at sites was placed into sizing the two and dividing up
IO intelligently to maximize IO throughput.

By comparison, Venti and the fossil filesystem built on top of
it would rightly be considered rather primitive versus zfs et
al.  But the Plan 9 infrastructure is (in many ways) much older,
and was oriented towards research anyway.

	- Dan C.

Back to comp.lang.c | Previous | Next — Previous in thread | Next in thread | Find similar

Thread

[sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-21 11:50 -0800
  Re: [sot] file times / file metadata "Pascal J. Bourguignon" <pjb@informatimago.com> - 2017-11-21 21:05 +0100
    Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-22 09:15 +1300
    Re: [sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-21 12:23 -0800
      Re: [sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-21 12:42 -0800
        Re: [sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-21 13:01 -0800
        Re: [sot] file times / file metadata "Pascal J. Bourguignon" <pjb@informatimago.com> - 2017-11-21 22:04 +0100
          Re: [sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-21 13:17 -0800
          Re: [sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-21 13:22 -0800
            Re: [sot] file times / file metadata "Pascal J. Bourguignon" <pjb@informatimago.com> - 2017-11-21 23:23 +0100
              Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-22 11:34 +1300
              Re: [sot] file times / file metadata Lew Pitcher <lew.pitcher@digitalfreehold.ca> - 2017-11-21 17:34 -0500
              Re: [sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-21 16:16 -0800
                Re: [sot] file times / file metadata "Pascal J. Bourguignon" <pjb@informatimago.com> - 2017-11-22 01:44 +0100
                Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-22 14:10 +1300
                Re: [sot] file times / file metadata gordonb.eiy78@burditt.org (Gordon Burditt) - 2017-11-28 01:55 -0600
          Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-22 11:07 +0100
            Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-22 09:13 -0500
              Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-22 15:50 +0100
                Re: [sot] file times / file metadata jameskuyper@verizon.net - 2017-11-22 07:08 -0800
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-22 16:22 +0100
                Re: [sot] file times / file metadata bartc <bc@freeuk.com> - 2017-11-22 15:37 +0000
                Re: [sot] file times / file metadata bartc <bc@freeuk.com> - 2017-11-22 15:43 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-22 15:01 -0500
                Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-23 17:52 +1300
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-23 14:12 -0500
                Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-24 08:28 +1300
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-24 08:52 -0500
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-23 09:39 +0100
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-23 14:17 -0500
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-11-28 16:44 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-28 14:23 -0500
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-11-28 19:35 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-30 15:54 -0500
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-11-30 21:43 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-04 11:42 -0500
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-12-04 19:35 +0000
                Re: [sot] file times / file metadata Keith Thompson <kst-u@mib.org> - 2017-12-04 11:53 -0800
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-12-04 20:11 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-04 15:03 -0500
                Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-11-28 19:42 +0000
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-11-28 20:13 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-30 15:57 -0500
                Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-11-30 21:17 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-04 11:56 -0500
                Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-12-04 18:36 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-04 15:07 -0500
                Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-12-04 22:27 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-04 17:57 -0500
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-12-05 08:50 +0100
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-05 08:49 -0500
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-12-05 15:17 +0100
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-05 16:32 -0500
                Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-12-05 15:29 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-05 16:37 -0500
                Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-12-06 01:26 +0000
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-12-06 12:38 -0500
                Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-12-06 01:27 +0000
                Re: [sot] file times / file metadata jameskuyper@verizon.net - 2017-11-28 09:35 -0800
                Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-29 08:24 +1300
                Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-29 08:27 +1300
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-28 14:29 -0500
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-29 09:51 +0100
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-30 15:59 -0500
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-29 10:00 +0100
                Re: [sot] file times / file metadata "James R. Kuyper" <jameskuyper@verizon.net> - 2017-11-29 09:11 -0500
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-29 15:28 +0100
                Re: [sot] file times / file metadata "James R. Kuyper" <jameskuyper@verizon.net> - 2017-11-29 11:32 -0500
                Re: [sot] file times / file metadata James Kuyper <jameskuyper@verizon.net> - 2017-11-23 15:05 -0500
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-24 10:20 +0100
                Re: [sot] file times / file metadata Jerry Stuckle <jstucklex@attglobal.net> - 2017-11-24 09:05 -0500
                Re: [sot] file times / file metadata Robert Wessel <robertwessel2@yahoo.com> - 2017-11-24 22:50 -0600
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-11-28 16:59 +0000
                Re: [sot] file times / file metadata Ian Collins <ian-news@hotmail.com> - 2017-11-29 08:26 +1300
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-29 10:10 +0100
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-11-29 21:01 +0000
                Re: [sot] file times / file metadata David Brown <david.brown@hesbynett.no> - 2017-11-29 23:13 +0100
                Re: [sot] file times / file metadata cross@spitfire.i.gajendra.net (Dan Cross) - 2017-11-30 17:26 +0000
              Re: [sot] file times / file metadata fir <profesor.fir@gmail.com> - 2017-11-22 08:00 -0800
              Re: [sot] file times / file metadata "Pascal J. Bourguignon" <pjb@informatimago.com> - 2017-11-22 17:11 +0100
            Re: [sot] file times / file metadata "Pascal J. Bourguignon" <pjb@informatimago.com> - 2017-11-22 16:51 +0100
              Re: [sot] file times / file metadata scott@slp53.sl.home (Scott Lurndal) - 2017-11-22 16:36 +0000

csiph-web