Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #231865 > unrolled thread

Question regarding hardware choices

Back to article view | Back to linux.debian.user

Contents

  Question regarding hardware choices Semih Ozlem <semihozlemlinuxuser@gmail.com> - 2021-02-19 16:50 +0100
    Re: Question regarding hardware choices Darac Marjal <mailinglist@darac.org.uk> - 2021-02-19 17:10 +0100
      Re: Question regarding hardware choices Nicolas George <george@nsup.org> - 2021-02-19 17:20 +0100
    Re: Question regarding hardware choices Liam Proven <lproven@gmail.com> - 2021-02-19 18:30 +0100
      Re: Question regarding hardware choices Brian <ad44@cityscape.co.uk> - 2021-02-19 21:00 +0100
        Re: Question regarding hardware choices Stefan Monnier <monnier@iro.umontreal.ca> - 2021-02-19 21:30 +0100
    Re: Question regarding hardware choices Linux-Fan <Ma_Sys.ma@web.de> - 2021-02-19 19:00 +0100
    Re: Question regarding hardware choices David Christensen <dpchrist@holgerdanske.com> - 2021-02-19 21:40 +0100
      Re: Question regarding hardware choices Darac Marjal <mailinglist@darac.org.uk> - 2021-02-19 21:50 +0100
        Re: Question regarding hardware choices David Christensen <dpchrist@holgerdanske.com> - 2021-02-19 22:50 +0100
          Re: Question regarding hardware choices Michael Stone <mstone@debian.org> - 2021-02-19 23:10 +0100
            Re: Question regarding hardware choices David Christensen <dpchrist@holgerdanske.com> - 2021-02-20 03:30 +0100
              The Elephant and the Philospher [was: Question regarding hardware  choices] <tomas@tuxteam.de> - 2021-02-20 10:00 +0100

#231865 — Question regarding hardware choices

[Multipart message — attachments visible in raw view] — view raw

Hello everyone,

In comparing performance what are the pros and cons to using

(i) live usb flash disk
(ii) live usb with persistence on a flash disk
(iii) full installation on a flash disk
(iv) full installation on an external hard disk (ssd or other)
(v) full installation on an internal hard disk (ssd or other)

Thanks

Semih

[toc] | [next] | [standalone]

#231866

[Multipart message — attachments visible in raw view] — view raw

On 19/02/2021 15:44, Semih Ozlem wrote:
> Hello everyone,
>
> In comparing performance what are the pros and cons to using
>
> (i) live usb flash disk
Usually, this is implemented with a compressed, read-only file system
which is read into RAM. "Live" Operating Systems developed from the era
of CD-ROMs etc, where the OS installation is physically read-only, so
any changes *have* to be kept in RAM. This will be relatively slow to
boot, and to load applications but once things are in RAM, they should
perform very well.
> (ii) live usb with persistence on a flash disk
This would be expected to perform almost identically to option (i). The
main advantage here is the ability to save files. Persistence won't
increase performance (that is, don't expect any sort of MRU caching or
similar).
> (iii) full installation on a flash disk

Assuming that we're talking a USB flash disk, this could either be
higher or lower performance than the Live installation. The Live
installation will have a compressed (or at least solid) image to read
from, whereas the full installation will have individual files to read.
Due to the almost non-existent seek times of flash memory, several
individual files are as easy to read as one big file of the same size,
but note that by uncompressing the Live installation you are perforce
requiring more data to be transferred off the disk. Most USB flash disks
(even USB3 drives) are relatively slow at transferring data (they
optimize for cost). Some devices DO have good controllers though, so YMMV.

> (iv) full installation on an external hard disk (ssd or other)

This probably depends much more on the actual disk, and its connection
to the host. Generally speaking, even a relatively basic spinning rust
disk will feel much more performant than an Flash Disk. Similarly, a
SATA SSD will feel more performant than a HDD. In terms of connection...
well there's so many to choose from: SCSI, USB, eSATA, Fibre Channel...
and so on and so on. Generally speaking, go for a protocol that's
designed for storage rather than general purpose.


> (v) full installation on an internal hard disk (ssd or other)

And here again, your choice of connection matters, too. SCSI, IDE, SATA,
SAS, NVMe. However, internal connectors usually have the advantage that
they're designed for short-range transport inside a metal box, so they
_tend_ to be faster than their external equivalents where you perhaps
need to handle a higher amount of noise.

>
> Thanks
>
> Semih

[toc] | [prev] | [next] | [standalone]

#231867

[Multipart message — attachments visible in raw view] — view raw

Darac Marjal (12021-02-19):
>						      Most USB flash disks
> (even USB3 drives) are relatively slow at transferring data (they
> optimize for cost). Some devices DO have good controllers though, so YMMV.

There is a more subtle problem here. Most USB flash disks will freeze
when blanking flash pages, that makes the whole system unresponsive.

A live distribution plus persistence is optimized to make a lot fewer
writes than a normal system, by mounting tmpfs over some variable disks,
and by not updating atime on the bulk of the system.

Regards,

-- 
  Nicolas George

[toc] | [prev] | [next] | [standalone]

#231868

On Fri, 19 Feb 2021 at 16:46, Semih Ozlem <semihozlemlinuxuser@gmail.com> wrote:
>
> Hello everyone,
>
> In comparing performance what are the pros and cons to using
>
> (i) live usb flash disk

Works, boots on anything (BIOS and UEFI), but you can't update and you
can't readily save your data. Boot speed slow, because the OS is in a
compressed disk image.

> (ii) live usb with persistence on a flash disk

Much the same as above. You can just save your data and settings, but
updating is very inefficient as you can't rewrite the contents of the
compressed disk image -- updated files go into your persistence
volume. OK for emergency use, not for routine operation.

> (iii) full installation on a flash disk

I have done this and used such systems.

If you want to use it on multiple different computers you may have
issues -- e.g. the same bootable key may not boot both a BIOS PC and a
UEFI PC. If there are OSes installed on the HDD as well, and you do an
update, then those OSes will be added to the GRUB menu on the key,
creating entries that won't work on any other PC.

If you always use it on the same machine, it's acceptable. I had a
company which sold such a bootable key for a while. It worked and the
speed is quite reasonable... but when you run normally, it is writing
settings to the key all the time.  When you update, it writes a lot of
data to the key. This wears out the flash memory on the key and in
time the key will fail. If in daily use, it may well wear out in 6
months or so. And when flash memory wears out, it is not like a
defective hard disk, which dies slowly and gives you a chance to
recover your data... when flash wears out, one failed write might
corrupt the volume and the whole disk stops working completely. I.e.
you lose all your data.

It works but it is only suitable for occasional or lightweight use. I
wrote a how-to guide for building such a system; it is here:
https://liam-on-linux.livejournal.com/50416.html
It is a little dated now but still valid. It details some
optimisations you can do to boost life -- such as using a filesystem
without journalling, disabling access-time storage and things. These
help but not much.

I have a fileserver that boots off a USB key, but it has no GUI and is
never normally used interactively. That works fine. Data is stored on
a RAID of SATA drives.


> (iv) full installation on an external hard disk (ssd or other)

So long as your computer boots happily from it, works fine. The same
caveats about about dual-booting and transportability mentioned above
apply.

Hard disks are a little slower than a USB 2 flash drive, but they
don't wear out. A USB3 flash drive might well be a lot quicker than a
hard disk,

An internal SSD is much faster than a USB2 flash drive and a little
quicker than a USB 3 external flash drive.

> (v) full installation on an internal hard disk (ssd or other)

This is the best  and fastest. Linux does not use licence keys,
activation or anything; a bootable HDD or SSD can be moved to another
computer without problems. (With a few obvious limitations; e.g. a
64-bit install will not work on a 32-bit computer.)

My recommendation is, have _both_ a small SSD and a big HDD. Install
with / (the root partition) on the HDD, and /home (the home partition)
and swap on the HDD. This gives you very fast boot times, very fast
program loading, etc. -- but lots of space on a medium that won't wear
out. User configuration files are relatively tiny and you will not
notice any slow-down in performance from having them on a slower HDD
compared to on a fast SDD.

-- 
Liam Proven – Profile: https://about.me/liamproven
Email: lproven@cix.co.uk – gMail/gTalk/gHangouts: lproven@gmail.com
Twitter/Facebook/LinkedIn/Flickr: lproven – Skype: liamproven
UK: +44 7939-087884 – ČR (+ WhatsApp/Telegram/Signal): +420 702 829 053

[toc] | [prev] | [next] | [standalone]

#231870

On Fri 19 Feb 2021 at 18:11:02 +0100, Liam Proven wrote:

[...]

> If you want to use it on multiple different computers you may have
> issues -- e.g. the same bootable key may not boot both a BIOS PC and a
> UEFI PC. If there are OSes installed on the HDD as well, and you do an
> update, then those OSes will be added to the GRUB menu on the key,
> creating entries that won't work on any other PC.

   GRUB_DISABLE_OS_PROBER=true

in /etc/default/grub does wonders for that.
> 
> If you always use it on the same machine, it's acceptable. I had a
> company which sold such a bootable key for a while. It worked and the
> speed is quite reasonable... but when you run normally, it is writing
> settings to the key all the time.  When you update, it writes a lot of
> data to the key. This wears out the flash memory on the key and in
> time the key will fail. If in daily use, it may well wear out in 6
> months or so. And when flash memory wears out, it is not like a
> defective hard disk, which dies slowly and gives you a chance to
> recover your data... when flash wears out, one failed write might
> corrupt the volume and the whole disk stops working completely. I.e.
> you lose all your data.

Key wear is not something I have encountered. Yet.

> It works but it is only suitable for occasional or lightweight use. I
> wrote a how-to guide for building such a system; it is here:
> https://liam-on-linux.livejournal.com/50416.html
> It is a little dated now but still valid. It details some
> optimisations you can do to boost life -- such as using a filesystem
> without journalling, disabling access-time storage and things. These
> help but not much.

I do all of that, plus having logs in memory. I live in hope of
a long life. 

[...]

-- 
Brian.

[toc] | [prev] | [next] | [standalone]

#231871

>> It works but it is only suitable for occasional or lightweight use.  I
>> wrote a how-to guide for building such a system; it is here:
>> https://liam-on-linux.livejournal.com/50416.html
>> It is a little dated now but still valid.  It details some
>> optimisations you can do to boost life -- such as using a filesystem
>> without journalling, disabling access-time storage and things. These
>> help but not much.
> I do all of that, plus having logs in memory. I live in hope of
> a long life. 

[ Personally I use `ext4` and I'm not convinced it performs
  significantly more writes than `ext2` in most usage patterns
  (obviously, in theory there's more, but in practice the use of a log
  allows delaying many other writes, many of which can end up cancel'd
  before they've had a chance to be performed).  ]

You may also want to make sure `/tmp` is a tmpfs.

Furthermore, you may want to check what's being written to your disk
by enabling /proc/sys/vm/block_dump and then looking at your log
(obviously, follow the advice above *before*).  You may find specific
applications that write data "needlessly" every once in a while (I used
to symlink some firefox files to /tmp for that reason).


        Stefan

[toc] | [prev] | [next] | [standalone]

#231869

[Multipart message — attachments visible in raw view] — view raw

TL;DR: Full installation on internal SSD is fastest,
       live (with or without persistence) is slower.

Semih Ozlem writes:

> Hello everyone,
>
> In comparing performance what are the pros and cons to using

What kind of performance do you look for?

 * CPU performance will mostly be equal for any of the methods

 * For things like network and GPU the properly installed systems
   (independent of device type used) can be faster due to the
   ability to install proprietary drivers. E.g. if you have an
   NVidia graphics chipset and install the proprietary driver,
   3D performance improves.

 * For the speed related to the boot media, there are at least
   these scenarios to consider:

    - system startup
    - installing system updates
    - running applications

> (i) live usb flash disk
> (ii) live usb with persistence on a flash disk
> (iii) full installation on a flash disk
> (iv) full installation on an external hard disk (ssd or other)
> (v) full installation on an internal hard disk (ssd or other)

>From slowest to fastest, I'd order your choices as follows:

system startup (slow -> fast)
(ii), (i), (iii), (iv), (v)

 * persistence slows down things due to additional mount
 * live systems take longer to boot than regular ones
   (less parallelism, automated user creation etc.)
 * (iii) - (v) are highly dependent on how fast the actual
   storage device is. An external NVMe SSD at USB 3.0 can
   easily beat an internal PATA HDD (just to have two extremes :) ).

installalling system updates
(i), (ii), (iii), (iv), (v)

 * installing updates to live systems is slow because it
   requires a complete re-image (difficult to do from within the system)
 * installing updates with persistence incurs some performance
   penalties (during and after installation) depending on how it
   is implemented
 * for (iii) - (v) it is again dependent on the actual hardware

running applications
(i), (ii), (iii), (iv), (v)

 * installed systems are faster because they need not
   re-create any of the application's directories etc. after
   first use of the respective application.
   Additionally, there is no need to use RAM for all of the
   current changes to the system state -- things like configuration
   files and installed upates are written back to the storage drive.
   As a result, more RAM is available for caching purposes and
   applications.

 * persistence may be faster than without because you can avoid
   re-installing applications manually.

 * There is one special case: Some live systems can be booted `toram`
   copying the whole image into RAM. As RAM is faster than the persistent
   storage options, this may have the best performance in certain cases.
   Hence, one could also order this (ii), (iii), (iv), (v), (i) with
   (i) having the "top performance".

Just my take on this, other good opinions exist, just see the other   
replies :)

HTH
Linux-Fan

PS: I had some issues sending this, I hope it arrives threaded correctly.

öö

[toc] | [prev] | [next] | [standalone]

#231872

On 2021-02-19 07:44, Semih Ozlem wrote:
> Hello everyone,
> 
> In comparing performance what are the pros and cons to using
> 
> (i) live usb flash disk
> (ii) live usb with persistence on a flash disk
> (iii) full installation on a flash disk
> (iv) full installation on an external hard disk (ssd or other)
> (v) full installation on an internal hard disk (ssd or other)

What network(s)?

What computer(s)?  CPU?  RAM?  Storage?  Network interfaces?  User 
interfaces?

Which version of Debian?

What services?  What applications?

"Performance" implies metrics.  What measurements interest you?  What is 
their priority?

Any other requirements?


David

[toc] | [prev] | [next] | [standalone]

#231873

[Multipart message — attachments visible in raw view] — view raw

On 19/02/2021 20:37, David Christensen wrote:
> On 2021-02-19 07:44, Semih Ozlem wrote:
>> Hello everyone,
>>
>> In comparing performance what are the pros and cons to using
>>
>> (i) live usb flash disk
>> (ii) live usb with persistence on a flash disk
>> (iii) full installation on a flash disk
>> (iv) full installation on an external hard disk (ssd or other)
>> (v) full installation on an internal hard disk (ssd or other)
>
> What network(s)?
Why would choice of network affect disk performance?
>
> What computer(s)?  CPU?  RAM?  Storage?  Network interfaces?  User
> interfaces?
Almost any CPU build this century is going to be able to keep up with
mass storage. I really don't see how any of these will materially affect
disk performance.
>
> Which version of Debian?
What improvements in Debian do you feel are relevant here? The switch to
libata?
>
> What services?  What applications?
>
> "Performance" implies metrics.  What measurements interest you?  What
> is their priority?
>
> Any other requirements?
>
>
> David
>

[toc] | [prev] | [next] | [standalone]

#231874

On 2021-02-19 12:43, Darac Marjal wrote:
> 
> On 19/02/2021 20:37, David Christensen wrote:
>> On 2021-02-19 07:44, Semih Ozlem wrote:
>>> Hello everyone,
>>>
>>> In comparing performance what are the pros and cons to using
>>>
>>> (i) live usb flash disk
>>> (ii) live usb with persistence on a flash disk
>>> (iii) full installation on a flash disk
>>> (iv) full installation on an external hard disk (ssd or other)
>>> (v) full installation on an internal hard disk (ssd or other)
>>
>> What network(s)?
> Why would choice of network affect disk performance?
>> What computer(s)?  CPU?  RAM?  Storage?  Network interfaces?  User
>> interfaces?
> Almost any CPU build this century is going to be able to keep up with
> mass storage. I really don't see how any of these will materially affect
> disk performance.
>>
>> Which version of Debian?
> What improvements in Debian do you feel are relevant here? The switch to
> libata?
>>
>> What services?  What applications?
>>
>> "Performance" implies metrics.  What measurements interest you?  What
>> is their priority?
>>
>> Any other requirements?


https://en.wikipedia.org/wiki/Blind_men_and_an_elephant


David

[toc] | [prev] | [next] | [standalone]

#231875

On Fri, Feb 19, 2021 at 01:45:23PM -0800, David Christensen wrote:
>https://en.wikipedia.org/wiki/Blind_men_and_an_elephant

You thought this was helpful how?

[toc] | [prev] | [next] | [standalone]

#231878

On 2021-02-19 13:55, Michael Stone wrote:
> On Fri, Feb 19, 2021 at 01:45:23PM -0800, David Christensen wrote:
>> https://en.wikipedia.org/wiki/Blind_men_and_an_elephant
> 
> You thought this was helpful how?

I frequently see a pattern on this list:

1.  Someone posts an interesting question, but provides little or no 
context that frames the question.

2.  Readers respond, making assumptions, stating answers framed within 
those assumptions, etc..

3.  Readers respond to responses, debating assumptions, debating 
answers, etc..

4.  The OP gets buried under a pile of responses that the OP may or may 
not understand or find useful.


So, metaphorically speaking, rather than blindly groping some portion of 
an unknown beast and attempting to describe it, I asked for a picture of 
the whole beast and a glimpse of its surroundings.  If and when the OP 
provides such information, readers will be better able to provide 
understandable and useful answers; thereby increasing the value of this 
list.


David

[toc] | [prev] | [next] | [standalone]

#231880 — The Elephant and the Philospher [was: Question regarding hardware choices]

[Multipart message — attachments visible in raw view] — view raw

On Fri, Feb 19, 2021 at 06:22:31PM -0800, David Christensen wrote:

[...]

> So, metaphorically speaking, rather than blindly groping some
> portion of an unknown beast and attempting to describe it [...]

CONGRATS! You just managed to get hold of that philospher who
was gripping firmly the elephant's tail. No idea what (s)he
was doing there nor why the elephant  was tolerating it (well,
elephants are said to be tolerant and nice, perhaps it's that).

;-D

Cheers
 - t

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.user

csiph-web