Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #228786 > unrolled thread

Problem with /var/mail file > 2GB with pop3

Back to article view | Back to linux.debian.user

Contents

  Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-19 23:00 +0100
    Re: Problem with /var/mail file > 2GB with pop3 Andy Smith <andy@strugglers.net> - 2020-11-19 23:30 +0100
      Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-21 01:50 +0100
    Re: Problem with /var/mail file > 2GB with pop3 Ulf Volmer <u.volmer@u-v.de> - 2020-11-20 00:10 +0100
      Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-21 01:50 +0100
        Re: Problem with /var/mail file > 2GB with pop3 deloptes <deloptes@gmail.com> - 2020-11-21 09:00 +0100
          Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-21 22:40 +0100
            Re: Problem with /var/mail file > 2GB with pop3 deloptes <deloptes@gmail.com> - 2020-11-22 01:10 +0100
              Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-22 19:00 +0100
                Re: Problem with /var/mail file > 2GB with pop3 deloptes <deloptes@gmail.com> - 2020-11-22 21:10 +0100
                Re: Problem with /var/mail file > 2GB with pop3 Andy Smith <andy@strugglers.net> - 2020-11-22 22:40 +0100
                  Re: Problem with /var/mail file > 2GB with pop3 Dan Ritter <dsr@randomstring.org> - 2020-11-22 23:40 +0100
                Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-23 07:10 +0100
                  Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-23 10:30 +0100
                    Re: Problem with /var/mail file > 2GB with pop3 Joe <joe@jretrading.com> - 2020-11-23 11:30 +0100
                      Re: Problem with /var/mail file > 2GB with pop3 Sven Hartge <sven@svenhartge.de> - 2020-11-23 12:20 +0100
                        Re: Problem with /var/mail file > 2GB with pop3 rhkramer@gmail.com - 2020-11-23 13:30 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 Didar Hossain <didar@purlo.in> - 2020-11-23 14:40 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 Sven Hartge <sven@svenhartge.de> - 2020-11-23 15:40 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 Andrei POPESCU <andreimpopescu@gmail.com> - 2020-11-23 16:40 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 Celejar <celejar@gmail.com> - 2020-11-23 19:30 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 Curt <curty@free.fr> - 2020-11-24 18:40 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-25 16:40 +0100
                              Re: Problem with /var/mail file > 2GB with pop3 Curt <curty@free.fr> - 2020-11-25 19:20 +0100
                                Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-26 05:00 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-24 20:50 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 Greg Wooledge <wooledg@eeg.ccf.org> - 2020-11-24 21:10 +0100
    Re: Problem with /var/mail file > 2GB with pop3 황병희 <soyeomul@doraji.xyz> - 2020-11-21 03:50 +0100
    Why use an email client AND sendmail/popa3d - trying to NOT hijack Keith Bainbridge <ke1thozgroups@gmx.com> - 2020-11-23 03:30 +0100
      Re: Why use an email client AND sendmail/popa3d - trying to NOT  hijack Joe <joe@jretrading.com> - 2020-11-23 11:50 +0100
        Re: Why use an email client AND sendmail/popa3d - trying to NOT hijack The Wanderer <wanderer@fastmail.fm> - 2020-11-23 12:20 +0100

Page 2 of 2 — ← Prev page 1 [2]

#228972

On Mon, 23 Nov 2020 17:38:19 +0200
Andrei POPESCU <andreimpopescu@gmail.com> wrote:

> On Lu, 23 nov 20, 07:24:37, rhkramer@gmail.com wrote:
> > On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
> > > Joe <joe@jretrading.com> wrote:
> > > > That's why we have IMAP, which doesn't use mbox.
> > > 
> > > The IMAP protocal and the backend storage have no connection.
> > 
> > Well, they do in a way -- if you use IMAP from your ISP for example, you don't 
> > need local storage on your email device (not any of mbox, maildir, or 
> > whatever).
> > 
> > (I don't use IMAP (I use POP3), but I assume that if I were using IMAP as 
> > described, I could save emails on my local device, but I'm not clear on that 
> > mechanism / storage method.)
>  
> It depends on the software used to retrieve the e-mails (via IMAP), e.g.  
> as far as I recall getmail can be configured to use either mbox or 
> Maildir as local storage.

Correct:

http://pyropus.ca/software/getmail/configuration.html#running-mda

Celejar

[toc] | [prev] | [next] | [standalone]

#229019

On 2020-11-23, rhkramer@gmail.com <rhkramer@gmail.com> wrote:
> On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
>> Joe <joe@jretrading.com> wrote:
>> > That's why we have IMAP, which doesn't use mbox.
>> 
>> The IMAP protocal and the backend storage have no connection.
>
> Well, they do in a way -- if you use IMAP from your ISP for example, you don't 
> need local storage on your email device (not any of mbox, maildir, or 
> whatever).
>
> (I don't use IMAP (I use POP3), but I assume that if I were using IMAP as 
> described, I could save emails on my local device, but I'm not clear on that 
> mechanism / storage method.)

The IMAP protocol and the local storage format are orthogonal, I believe is what
he said, so you've come full circle in your reflections.

[toc] | [prev] | [next] | [standalone]

#229066

On Tue 24 Nov 2020 at 17:17:24 (+0000), Curt wrote:
> On 2020-11-23, rhkramer@gmail.com <rhkramer@gmail.com> wrote:
> > On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
> >> Joe <joe@jretrading.com> wrote:
> >> > That's why we have IMAP, which doesn't use mbox.
> >> 
> >> The IMAP protocal and the backend storage have no connection.
                               ↑↑↑↑↑↑↑↑↑↑↑↑↑↑↑
> >
> > Well, they do in a way -- if you use IMAP from your ISP for example, you don't 
> > need local storage on your email device (not any of mbox, maildir, or 
> > whatever).
> >
> > (I don't use IMAP (I use POP3), but I assume that if I were using IMAP as 
> > described, I could save emails on my local device, but I'm not clear on that 
> > mechanism / storage method.)
> 
> The IMAP protocol and the local storage format are orthogonal, I believe is what
> he said, so you've come full circle in your reflections.

AIUI the backend storage is what the IMAP (and POP) protocols serve,
and the local storage is where the emails are stored when they have
been served.

It appears that the OP, like me, has their emails served by a remote
server, so one typically has no control over, knowledge or concern
about the hosting service's backend storage (beyond its being
reliable, 24/7 available, etc).

There's a big difference between the requirements for your local
storage when using POP compared with IMAP. When using POP in a
conventional manner (transfer, and delete at the server end), you
need reliable local storage. And you also need reliable file locking
at least when you're using mbox files.

OTOH for IMAP, there's no requirement for any local storage at all,
beyond a screen display so that you can read the emails. The client
*can* cache emails and/or make local copies, but it's not necessary
because you can reacquire any email at will.

Joe's IMAP *server* does have backend storage, but it looks as though
that's being fed by SMTP, which AIUI is not what the OP is thinking
of using (nor me).

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#229071

On 2020-11-25, David Wright <deblis@lionunicorn.co.uk> wrote:
>
> There's a big difference between the requirements for your local
> storage when using POP compared with IMAP. When using POP in a
> conventional manner (transfer, and delete at the server end), you
> need reliable local storage. And you also need reliable file locking
> at least when you're using mbox files.

The requirements may be different, but here, using Alpine, I think the
local storage format is identical and unrelated to the protocol used
on the server end (POP or IMAP). 

[toc] | [prev] | [next] | [standalone]

#229097

On Wed 25 Nov 2020 at 17:56:27 (+0000), Curt wrote:
> On 2020-11-25, David Wright <deblis@lionunicorn.co.uk> wrote:
> >
> > There's a big difference between the requirements for your local
> > storage when using POP compared with IMAP. When using POP in a
> > conventional manner (transfer, and delete at the server end), you
> > need reliable local storage. And you also need reliable file locking
> > at least when you're using mbox files.
> 
> The requirements may be different, but here, using Alpine, I think the
> local storage format is identical and unrelated to the protocol used
> on the server end (POP or IMAP). 

So it should be, assuming "here" means on your local computer systems.
But I'm not talking about which form of local storage you've chosen to
use, nor even which forms are available to you.

Perhaps I should be more specific about what I mean by "requirements".
I don't mean required by the protocol, nor the computer system, nor
anything else, but by any sensible user making serious use of emails
for more than trivia.

Using IMAP, it's an easy matter to make sure that an email is secure
and backed up before you delete it from the server. Using POP in a
conventional manner (deliberately or involuntarily), that's more
difficult. My requirements would be a RAID filesystem, UPS, and online
backup were I forced to use POP.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#229033

On Mon 23 Nov 2020 at 07:24:37 (-0500), rhkramer@gmail.com wrote:
> On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
> > Joe <joe@jretrading.com> wrote:
> > > That's why we have IMAP, which doesn't use mbox.
> > 
> > The IMAP protocal and the backend storage have no connection.

I didn't think this thread was about the backend at all, but about
where to place the emails being fetched, and hence the protocol
to use.

I don't think there's much concern about the final destination of the
emails either, but only the immediate destination, ie where emails are
placed at the moment of first arrival on the OP's system.

> Well, they do in a way -- if you use IMAP from your ISP for example, you don't 
> need local storage on your email device (not any of mbox, maildir, or 
> whatever).
> 
> (I don't use IMAP (I use POP3), but I assume that if I were using IMAP as 
> described, I could save emails on my local device, but I'm not clear on that 
> mechanism / storage method.)

It may vary by client. With mutt over IMAP, opening a remote INBOX
will synchronise the INBOX's headers (only) with a local cache.¹
If and when you read a new message, mutt reads it into a cache
where each email is a separate file. That cache is to avoid having
to re-transfer the email each time you open it in your INBOX.

When you save an email on your computer, it's copied into whichever
type of storage you prefer. It might be a mailbox, meaning an mbox
or maildir (depending on how you configured it), or just an ordinary
file in the filesystem.

The main difference between a cached email and one that's been saved
to a file or mbox is that the saved one has an extra line (acting
as a separator) appended at the start, and a blank line at the end.

Two big advantages of IMAP are:
. Having looked at the header, you can delete the email without
  having even transferred the body.
. While you read, save, back up, etc, your emails, they're still
  safely sitting on the server. To truly delete them, you have
  to flag them as deleted, and then synchronise with the server
  (ie open a new folder, or close the client, or explicitly
  synchronise).

Repeat, all this is for mutt/IMAP.

¹ Assuming you want a cache. IIRC this first cache is crudely
  managed, gently growing all the time, and pruned just by
  deleting it.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#229036

On Tue, Nov 24, 2020 at 01:44:28PM -0600, David Wright wrote:
> On Mon 23 Nov 2020 at 07:24:37 (-0500), rhkramer@gmail.com wrote:
> > On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
> > > Joe <joe@jretrading.com> wrote:
> > > > That's why we have IMAP, which doesn't use mbox.
> > > 
> > > The IMAP protocal and the backend storage have no connection.
> 
> I didn't think this thread was about the backend at all, but about
> where to place the emails being fetched, and hence the protocol
> to use.

IMAP isn't like fetchmail.  It isn't even like POP3.  It doesn't pull
all the email down to the local client system and store it on disk.

That's the POP3 model.

The IMAP model is that the client opens a persistent connection to the
IMAP server, and pulls down one message at a time, only when the client
wishes to read that particular message.  They stay on the server until
the client asks for one to be deleted.  The client does not, in general,
store a permanent copy of a message.

Look at the traditional email model first, with no POP3 or IMAP in
the picture.

Let's take two users, in two different organizations.  Traditionally
their names are Alice and Bob.  We'll say that Alice works at Foo Corp
(email domain foo.corp), and Bob is a student at Bar College (email
domain bar.college).  Foo Corp and Bar College both use a single Unix
server with local user accounts, running a traditional Unix MTA, with
mail delivered locally.

Alice writes a message to Bob (bob@bar.college).  After writing it and
hitting the send button, the message is queued up for delivery by
the MTA on her Unix system.

Her MTA looks up the MX record for bar.college in DNS, and gets the
name and IP address of Bob's Unix host.  The two MTAs talk to each
other, and the message is passed using the Internet.

Now, the MTA on Bob's server delivers the message locally.  It opens
Bob's mailbox (or Maildir), and writes the message there.  It calls
a "sync" function to make sure the bytes are actually written.  Now it's
stored safely on disk.  Bob's MTA tells Alice's MTA that everything is
OK.  Alice's MTA deletes its copy of the message from its queue, and
everything is happy.

At some point in the future, Bob will login to his account, and read
his mail, using some local MUA such as elm or mailx.  The message is
stored on the local disk, in whatever format his MTA uses.

Now, let's introduce the IMAP layer.  Instead of logging in directly
to the Unix server, let's say Bob is a Microsoft Windows user, and
doesn't know how to ssh into his mail server.  So, Bob's administrator
sets up an IMAP server on the Unix host.

Now, the IMAP server can read Bob's mailbox (in whatever format), and
present the message to Bob's IMAP client.  Bob can read the messages,
reply to them, delete them, and so on.  All the things that email users
typically do.

The MTA is responsible for the local deliveries and storage of the
messages that are received.  The IMAP server is responsible for
presenting the already-stored messages to the IMAP client, which is
responsible for interacting with the human user.

As long as the MTA, the IMAP server (if any), and the local MUAs (if
any) all agree on the mailbox location and format, everything works.

The mbox formats store all messages in a single gigantic file.  Each
message's start is indicated by the presence of a specifc string at
the start of a line.  When the MTA delivers a message, it simply appends
to the end of the mbox file.  When an MUA or IMAP server wants to delete
a message from the mailbox, it has to copy out the entire mailbox, minus
the message that was deleted.  It has to try to do this in a way that
won't explode if the MTA is trying to deliver a message at the same
moment.

In general, this is hard, involves file locking, and sucks a lot.

The Maildir format stores each message in a separate file.  When the MTA
wants to deliver a new message, it opens a new file (with a name chosen
according to heuristics that make it extremely unlikely to be used by
two different programs at the same time), writes the message there, and
closes it.  When an MUA or IMAP server wants to delete an existing message,
it simply removes the file containing that message.  It doesn't have to
lock the entire mailbox, or make a copy of it.

There is a lot more detail involved, of course, but that's the gist of it.

[toc] | [prev] | [next] | [standalone]

#228826

Dear Flo,

Flo <debianflo@gmx.at> writes:

> Hi All,
>
> I am using Debian Buster, Thunderbird, Sendmail and popa3d to get emails.
>
> The mail files for each account are stored at /var/mail. No it has
> come to that point that such a file exceeded 2GB. And 'Get Messages'
> doesn't work anymore.
>
> Does anyone know about this issue? Any hints to solve it? I could try
> a different pop3 server?
>
> Any help is appreciated.

sorry man off-topic... so clever persons use Gmane for mailing
lists. and personal emails get by Google Apps or Gmail that have very
big file system for email messages.

Sincerely, off-topic man Byung-Hee

-- 
^고맙습니다 _和合團結_ 감사합니다_^))//

[toc] | [prev] | [next] | [standalone]

#228932 — Why use an email client AND sendmail/popa3d - trying to NOT hijack

[Multipart message — attachments visible in raw view] — view raw

Good afternon All

I was interested to read that Flo, the OP, uses separate mail
collection, sendmail and thunderbird. Some of the replies sound like
this is a common practice.

What are the advantages of this set of processes over letting tbird do
it all? - or any other client for that matter?

Would it save me from my fairly regular 'can't find profile' errors?

Thanks

PS  Am I wrong to avoid 'everyting in 1 file' where possible (mail dir
rather than mbox in this case)? OK this is probably a whole separate topic.

--
Keith Bainbridge

ke1thozgroups@gmx.com




-------- Forwarded Message --------
Subject: 	Problem with /var/mail file > 2GB with pop3
Resent-Date: 	Thu, 19 Nov 2020 21:52:35 +0000 (UTC)
Resent-From: 	debian-user@lists.debian.org
Date: 	Thu, 19 Nov 2020 22:42:53 +0100
From: 	Flo <debianflo@gmx.at>
To: 	debian-user@lists.debian.org



Hi All,

I am using Debian Buster, Thunderbird, Sendmail and popa3d to get emails.


Any help is appreciated.

Thanks,
Flo

[toc] | [prev] | [next] | [standalone]

#228945 — Re: Why use an email client AND sendmail/popa3d - trying to NOT hijack

On Mon, 23 Nov 2020 13:21:25 +1100
Keith Bainbridge <ke1thozgroups@gmx.com> wrote:

> Good afternon All
> 
> I was interested to read that Flo, the OP, uses separate mail
> collection, sendmail and thunderbird. Some of the replies sound like
> this is a common practice.
> 
> What are the advantages of this set of processes over letting tbird do
> it all? - or any other client for that matter?

As far as I know, TB isn't an MTA, it can send email only as a client
to an MTA somewhere else. So it's not doing it all.

A lot depends how you want to send and receive emails. If you're using
an external email service, you can get away with just an email client,
or even use webmail. If you're sending and receiving yourself, you'll
need an MTA and an email distribution method such as POP3 or IMAP, as
well as clients on any devices you have. If you're also collecting
email from an external service, you'll need an email collector such as
fetchmail or procmail, to keep all email centrally stored. 
> 
> Would it save me from my fairly regular 'can't find profile' errors?
> 
> Thanks

Don't know, I gave up TB for Claws-mail long ago, TB was just too
painfully slow.

> 
> PS  Am I wrong to avoid 'everyting in 1 file' where possible (mail dir
> rather than mbox in this case)? OK this is probably a whole separate
> topic.

As I've posted elsewhere, I have about 3GB of email. I would not
consider putting that in one file.

-- 
Joe

[toc] | [prev] | [next] | [standalone]

#228946 — Re: Why use an email client AND sendmail/popa3d - trying to NOT hijack

[Multipart message — attachments visible in raw view] — view raw

On 2020-11-23 at 05:43, Joe wrote:

> On Mon, 23 Nov 2020 13:21:25 +1100 Keith Bainbridge
> <ke1thozgroups@gmx.com> wrote:

>> PS  Am I wrong to avoid 'everyting in 1 file' where possible (mail
>> dir rather than mbox in this case)? OK this is probably a whole
>> separate topic.
> 
> As I've posted elsewhere, I have about 3GB of email. I would not
> consider putting that in one file.

Speaking as a user of Thunderbird, I have ~20GB of E-mail (including
archives which date back well over a decade if not further), split
across a few accounts plus the "Local Folders" non-account.

It's divided into a total of 422 different mail-client-displayed
"folders" (although some of them are parent-folder only, they don't
contain actual messages), each of which is stored as a single file (not
mbox or similar, but the internal "Mork" database format, which as I
understand matters even Thunderbird may now be moving away from).

That averages out to ~47MB per file. After discounting the
otherwise-empty parent folders, the realistic figure is actually
probably somewhere in the 100MB-200MB range. When a given mailing list's
folder gets too large for my taste (or large enough that I start to
notice delays reading or writing that folder), I create a separate
"archive" folders for it by year, and move previous years' mail from
that folder into those per-year archive folders; this tends to happen
when the folder's contents reach somewhere between 10,000 and 20,000
messages.

This isn't necessarily a particularly ideal way of handling things, but
it's worked well for me thus far.

-- 
   The Wanderer

The reasonable man adapts himself to the world; the unreasonable one
persists in trying to adapt the world to himself. Therefore all
progress depends on the unreasonable man.         -- George Bernard Shaw

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.user

csiph-web