Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.user > #228786 > unrolled thread

Problem with /var/mail file > 2GB with pop3

Back to article view | Back to linux.debian.user

Contents

  Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-19 23:00 +0100
    Re: Problem with /var/mail file > 2GB with pop3 Andy Smith <andy@strugglers.net> - 2020-11-19 23:30 +0100
      Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-21 01:50 +0100
    Re: Problem with /var/mail file > 2GB with pop3 Ulf Volmer <u.volmer@u-v.de> - 2020-11-20 00:10 +0100
      Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-21 01:50 +0100
        Re: Problem with /var/mail file > 2GB with pop3 deloptes <deloptes@gmail.com> - 2020-11-21 09:00 +0100
          Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-21 22:40 +0100
            Re: Problem with /var/mail file > 2GB with pop3 deloptes <deloptes@gmail.com> - 2020-11-22 01:10 +0100
              Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-22 19:00 +0100
                Re: Problem with /var/mail file > 2GB with pop3 deloptes <deloptes@gmail.com> - 2020-11-22 21:10 +0100
                Re: Problem with /var/mail file > 2GB with pop3 Andy Smith <andy@strugglers.net> - 2020-11-22 22:40 +0100
                  Re: Problem with /var/mail file > 2GB with pop3 Dan Ritter <dsr@randomstring.org> - 2020-11-22 23:40 +0100
                Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-23 07:10 +0100
                  Re: Problem with /var/mail file > 2GB with pop3 Flo <debianflo@gmx.at> - 2020-11-23 10:30 +0100
                    Re: Problem with /var/mail file > 2GB with pop3 Joe <joe@jretrading.com> - 2020-11-23 11:30 +0100
                      Re: Problem with /var/mail file > 2GB with pop3 Sven Hartge <sven@svenhartge.de> - 2020-11-23 12:20 +0100
                        Re: Problem with /var/mail file > 2GB with pop3 rhkramer@gmail.com - 2020-11-23 13:30 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 Didar Hossain <didar@purlo.in> - 2020-11-23 14:40 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 Sven Hartge <sven@svenhartge.de> - 2020-11-23 15:40 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 Andrei POPESCU <andreimpopescu@gmail.com> - 2020-11-23 16:40 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 Celejar <celejar@gmail.com> - 2020-11-23 19:30 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 Curt <curty@free.fr> - 2020-11-24 18:40 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-25 16:40 +0100
                              Re: Problem with /var/mail file > 2GB with pop3 Curt <curty@free.fr> - 2020-11-25 19:20 +0100
                                Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-26 05:00 +0100
                          Re: Problem with /var/mail file > 2GB with pop3 David Wright <deblis@lionunicorn.co.uk> - 2020-11-24 20:50 +0100
                            Re: Problem with /var/mail file > 2GB with pop3 Greg Wooledge <wooledg@eeg.ccf.org> - 2020-11-24 21:10 +0100
    Re: Problem with /var/mail file > 2GB with pop3 황병희 <soyeomul@doraji.xyz> - 2020-11-21 03:50 +0100
    Why use an email client AND sendmail/popa3d - trying to NOT hijack Keith Bainbridge <ke1thozgroups@gmx.com> - 2020-11-23 03:30 +0100
      Re: Why use an email client AND sendmail/popa3d - trying to NOT  hijack Joe <joe@jretrading.com> - 2020-11-23 11:50 +0100
        Re: Why use an email client AND sendmail/popa3d - trying to NOT hijack The Wanderer <wanderer@fastmail.fm> - 2020-11-23 12:20 +0100

Page 1 of 2  [1] 2  Next page →

#228786 — Problem with /var/mail file > 2GB with pop3

Hi All,

I am using Debian Buster, Thunderbird, Sendmail and popa3d to get emails.

The mail files for each account are stored at /var/mail. No it has come 
to that point that such a file exceeded 2GB. And 'Get Messages' doesn't 
work anymore.

Does anyone know about this issue? Any hints to solve it? I could try a 
different pop3 server?

Any help is appreciated.

Thanks,
Flo

[toc] | [next] | [standalone]

#228787

Hello,

On Thu, Nov 19, 2020 at 10:42:53PM +0100, Flo wrote:
> The mail files for each account are stored at /var/mail. No it has come to
> that point that such a file exceeded 2GB. And 'Get Messages' doesn't work
> anymore.
> 
> Does anyone know about this issue? Any hints to solve it? I could try a
> different pop3 server?

I have no experience with popa3d but if it supports Maildir instead
of mbox format then you could start storing users' mails in Maildir
format. Maildir's one file per message approach usually provides
better performance, though you do end up with a lot of metadata
activity on the filesystem.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#228823

On 11/19/20 11:23 PM, Andy Smith wrote:
> Hello,
> 
> On Thu, Nov 19, 2020 at 10:42:53PM +0100, Flo wrote:
>> The mail files for each account are stored at /var/mail. No it has come to
>> that point that such a file exceeded 2GB. And 'Get Messages' doesn't work
>> anymore.
>>
>> Does anyone know about this issue? Any hints to solve it? I could try a
>> different pop3 server?
> 
> I have no experience with popa3d but if it supports Maildir instead
> of mbox format then you could start storing users' mails in Maildir
> format. Maildir's one file per message approach usually provides
> better performance, though you do end up with a lot of metadata
> activity on the filesystem.
> 

I don't think it can deal with the Maildir format. Thank you for sharing 
this idea.
Flo

[toc] | [prev] | [next] | [standalone]

#228788

On 19.11.20 22:42, Flo wrote:

> I am using Debian Buster, Thunderbird, Sendmail and popa3d to get emails.
> 
> The mail files for each account are stored at /var/mail. No it has come
> to that point that such a file exceeded 2GB. And 'Get Messages' doesn't
> work anymore.

This limit is defined in the source of popa3d.

ulf@deb10-p330:~/popa3d$ grep -r "#define MAX_MAILBOX_OPEN_BYTES" *
debian/README.Debian:#define MAX_MAILBOX_OPEN_BYTES          200000000
popa3d-1.0.3/params.h:#define MAX_MAILBOX_OPEN_BYTES           2147483647

I guess you have to recompile the package to change this.
But please consult README.Debian before.

Or (in my opinion the better solution) swicth to dovecot.

Best regards
Ulf

[toc] | [prev] | [next] | [standalone]

#228824

On 11/19/20 11:47 PM, Ulf Volmer wrote:
> On 19.11.20 22:42, Flo wrote:
> 
>> I am using Debian Buster, Thunderbird, Sendmail and popa3d to get emails.
>>
>> The mail files for each account are stored at /var/mail. No it has come
>> to that point that such a file exceeded 2GB. And 'Get Messages' doesn't
>> work anymore.
> 
> This limit is defined in the source of popa3d.
> 

Thank you, that was a good hint.

> ulf@deb10-p330:~/popa3d$ grep -r "#define MAX_MAILBOX_OPEN_BYTES" *
> debian/README.Debian:#define MAX_MAILBOX_OPEN_BYTES          200000000
> popa3d-1.0.3/params.h:#define MAX_MAILBOX_OPEN_BYTES           2147483647
> 
> I guess you have to recompile the package to change this.
> But please consult README.Debian before.
> 
> Or (in my opinion the better solution) swicth to dovecot.
> 

I installed dovecot but it didn't really work. Actually only with one 
specific account, the one which produces this big mbox file. And I 
couldn't figure out the reason.

So I recompiled popa3d and it seems to work now.

Thanks a lot.
Flo

> Best regards
> Ulf
> 

[toc] | [prev] | [next] | [standalone]

#228835

Flo wrote:

> I installed dovecot but it didn't really work. Actually only with one
> specific account, the one which produces this big mbox file. And I
> couldn't figure out the reason.
> 
> So I recompiled popa3d and it seems to work now.

Anyway - consider migrating from the mbox format to something else.

[toc] | [prev] | [next] | [standalone]

#228873

On Sat 21 Nov 2020 at 08:55:41 (+0100), deloptes wrote:
> Flo wrote:
> 
> > I installed dovecot but it didn't really work. Actually only with one
> > specific account, the one which produces this big mbox file. And I
> > couldn't figure out the reason.
> > 
> > So I recompiled popa3d and it seems to work now.
> 
> Anyway - consider migrating from the mbox format to something else.

I can't understand how anyone would want to have a live INBOX
file containing 2GB of emails. Apart from the risks, it just
seems so disorganised.

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#228891

David Wright wrote:

> I can't understand how anyone would want to have a live INBOX
> file containing 2GB of emails. Apart from the risks, it just
> seems so disorganised.

I don't even try - there are all kinds of people out there - and everybody
is free. And surprisingly the OP recompiles the package. In this case it is
enough to just point out the nonsense of the subject "Problem
with /var/mail file > 2GB with pop3".

[toc] | [prev] | [next] | [standalone]

#228925

On 2020-11-22 01:04, deloptes wrote:
> David Wright wrote:
>
>> I can't understand how anyone would want to have a live INBOX
>> file containing 2GB of emails. Apart from the risks, it just
>> seems so disorganised.
>
> I don't even try - there are all kinds of people out there - and everybody
> is free. And surprisingly the OP recompiles the package. In this case it is
> enough to just point out the nonsense of the subject "Problem
> with /var/mail file > 2GB with pop3".
>
>

Let's assume an average message size of 20MB. Then 100 messages are
enough to make it INBOX file that big. This doesn't necessarily mean
that this is disorganized.

Flo

[toc] | [prev] | [next] | [standalone]

#228928

Flo wrote:

> Let's assume an average message size of 20MB. Then 100 messages are
> enough to make it INBOX file that big. This doesn't necessarily mean
> that this is disorganized.

No, it means exactly this, because when mbox was in developed and actively
in use there was no message of 20MB in size.
Mailbox format was developed and even some database backends to store and
serve mails faster.
There are also tools that convert the one to the other. Consider moving
forward from where you are.

regards

[toc] | [prev] | [next] | [standalone]

#228929

Hi Flo,

On Sun, Nov 22, 2020 at 06:57:46PM +0100, Flo wrote:
> Let's assume an average message size of 20MB. Then 100 messages are
> enough to make it INBOX file that big. This doesn't necessarily mean
> that this is disorganized.

The average size of mail I have received in the last 2 months is
16.2KiB. It seems you receive mails one thousand times as large as I
do.

Cheers,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

[toc] | [prev] | [next] | [standalone]

#228930

Andy Smith wrote: 
> Hi Flo,
> 
> On Sun, Nov 22, 2020 at 06:57:46PM +0100, Flo wrote:
> > Let's assume an average message size of 20MB. Then 100 messages are
> > enough to make it INBOX file that big. This doesn't necessarily mean
> > that this is disorganized.
> 
> The average size of mail I have received in the last 2 months is
> 16.2KiB. It seems you receive mails one thousand times as large as I
> do.

Average size of my current inbox is almost 300KB, but this is massively
skewed because I have a bunch of large attachments hanging
around.

Average size of my archived inbox (i.e. no mailing
lists, post-filtering of various sorts) is just under 40KB.

In any case, 20MB seems... indicative of other problems.

-dsr-

[toc] | [prev] | [next] | [standalone]

#228939

On Sun 22 Nov 2020 at 18:57:46 (+0100), Flo wrote:
> On 2020-11-22 01:04, deloptes wrote:
> > David Wright wrote:
> >
> >> I can't understand how anyone would want to have a live INBOX
> >> file containing 2GB of emails. Apart from the risks, it just
> >> seems so disorganised.
> >
> > I don't even try - there are all kinds of people out there - and everybody
> > is free. And surprisingly the OP recompiles the package. In this case it is
> > enough to just point out the nonsense of the subject "Problem
> > with /var/mail file > 2GB with pop3".
> 
> Let's assume an average message size of 20MB. Then 100 messages are
> enough to make it INBOX file that big. This doesn't necessarily mean
> that this is disorganized.

Let's assume an average word length of 10 characters, completely excessive.
That would give an email of 2 million words. That would give you about
4000 pages of single-spaced text. No one writes emails like that;
not even Proust, were he alive.

So these hefty emails are really files being transferred as attachments,
aren't they? And having been transferred, one would usually archive
the files, and move the emails out of the live INBOX into other mail
files or directories.

Perhaps not. What would be the advantage of leaving 100 such sizeable
files in a live INBOX?

Cheers,
David.

[toc] | [prev] | [next] | [standalone]

#228943

On 11/23/20 7:05 AM, David Wright wrote:
> On Sun 22 Nov 2020 at 18:57:46 (+0100), Flo wrote:
>> On 2020-11-22 01:04, deloptes wrote:
>>> David Wright wrote:
>>>
>>>> I can't understand how anyone would want to have a live INBOX
>>>> file containing 2GB of emails. Apart from the risks, it just
>>>> seems so disorganised.
>>>
>>> I don't even try - there are all kinds of people out there - and everybody
>>> is free. And surprisingly the OP recompiles the package. In this case it is
>>> enough to just point out the nonsense of the subject "Problem
>>> with /var/mail file > 2GB with pop3".
>>
>> Let's assume an average message size of 20MB. Then 100 messages are
>> enough to make it INBOX file that big. This doesn't necessarily mean
>> that this is disorganized.
> 
> Let's assume an average word length of 10 characters, completely excessive.
> That would give an email of 2 million words. That would give you about
> 4000 pages of single-spaced text. No one writes emails like that;
> not even Proust, were he alive.
> 
> So these hefty emails are really files being transferred as attachments,
> aren't they? And having been transferred, one would usually archive
> the files, and move the emails out of the live INBOX into other mail
> files or directories.
> 
> Perhaps not. What would be the advantage of leaving 100 such sizeable
> files in a live INBOX?
> 

To have them (files and context) available on a other devices as well 
for a limited amount of time. Not only on the device where the files are 
archived.

Flo

[toc] | [prev] | [next] | [standalone]

#228944

On Mon, 23 Nov 2020 10:25:28 +0100
Flo <debianflo@gmx.at> wrote:

> On 11/23/20 7:05 AM, David Wright wrote:
> > On Sun 22 Nov 2020 at 18:57:46 (+0100), Flo wrote:  
> >> On 2020-11-22 01:04, deloptes wrote:  
> >>> David Wright wrote:
> >>>  
> >>>> I can't understand how anyone would want to have a live INBOX
> >>>> file containing 2GB of emails. Apart from the risks, it just
> >>>> seems so disorganised.  
> >>>
> >>> I don't even try - there are all kinds of people out there - and
> >>> everybody is free. And surprisingly the OP recompiles the
> >>> package. In this case it is enough to just point out the nonsense
> >>> of the subject "Problem with /var/mail file > 2GB with pop3".  
> >>
> >> Let's assume an average message size of 20MB. Then 100 messages are
> >> enough to make it INBOX file that big. This doesn't necessarily
> >> mean that this is disorganized.  
> > 
> > Let's assume an average word length of 10 characters, completely
> > excessive. That would give an email of 2 million words. That would
> > give you about 4000 pages of single-spaced text. No one writes
> > emails like that; not even Proust, were he alive.
> > 
> > So these hefty emails are really files being transferred as
> > attachments, aren't they? And having been transferred, one would
> > usually archive the files, and move the emails out of the live
> > INBOX into other mail files or directories.
> > 
> > Perhaps not. What would be the advantage of leaving 100 such
> > sizeable files in a live INBOX?
> >   
> 
> To have them (files and context) available on a other devices as well 
> for a limited amount of time. Not only on the device where the files
> are archived.

That's why we have IMAP, which doesn't use mbox.

I have about 3GB emails live on my IMAP server, accessible to all my
devices, that's about 20 years' worth.

-- 
Joe

[toc] | [prev] | [next] | [standalone]

#228947

Joe <joe@jretrading.com> wrote:

> That's why we have IMAP, which doesn't use mbox.

The IMAP protocal and the backend storage have no connection.

You can for example use mbox with Dovecot just fine. (Not that anyone
would *want* to, but that is a different story.)

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

[toc] | [prev] | [next] | [standalone]

#228951

On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
> Joe <joe@jretrading.com> wrote:
> > That's why we have IMAP, which doesn't use mbox.
> 
> The IMAP protocal and the backend storage have no connection.

Well, they do in a way -- if you use IMAP from your ISP for example, you don't 
need local storage on your email device (not any of mbox, maildir, or 
whatever).

(I don't use IMAP (I use POP3), but I assume that if I were using IMAP as 
described, I could save emails on my local device, but I'm not clear on that 
mechanism / storage method.)

[toc] | [prev] | [next] | [standalone]

#228956

On Mon, Nov 23, 2020 at 07:24:37AM -0500, rhkramer@gmail.com wrote:
> On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
> > Joe <joe@jretrading.com> wrote:
> > > That's why we have IMAP, which doesn't use mbox.
> > 
> > The IMAP protocal and the backend storage have no connection.
> 
> Well, they do in a way -- if you use IMAP from your ISP for example, you don't 
> need local storage on your email device (not any of mbox, maildir, or 
> whatever).
> 
> (I don't use IMAP (I use POP3), but I assume that if I were using IMAP as 
> described, I could save emails on my local device, but I'm not clear on that 
> mechanism / storage method.)

Thought I would just jump in.

Dovecot has "single instance attachment storage" (SIS) as well as its own native
mdbox binary format (multiple emails per file indexed efficiently).

The SIS is a feature that I am really excited about since we have multiple user
receiveing the same email - it really saves on storage space.

Kind regards,
Didar

-- 
"I can imagine no greater misfortune for a cultured people than to see in
 the hands of the rulers not only the civil, but also the religious power."
            [Caius Valerius Catullus, Roman poet 87-54 BC]

[toc] | [prev] | [next] | [standalone]

#228959

Didar Hossain <didar@purlo.in> wrote:

> Dovecot has "single instance attachment storage" (SIS) as well as its
> own native mdbox binary format (multiple emails per file indexed
> efficiently).

> The SIS is a feature that I am really excited about since we have
> multiple user receiveing the same email - it really saves on storage
> space.

Be careful with SIS in Dovecot. There be dragons!

There are some yet-to-be-solved problems with it, mostly in the "remove
attachments no longer needed" region.

Scour the list archives before using it.

Grüße,
Sven.

-- 
Sigmentation fault. Core dumped.

[toc] | [prev] | [next] | [standalone]

#228962

[Multipart message — attachments visible in raw view] — view raw

On Lu, 23 nov 20, 07:24:37, rhkramer@gmail.com wrote:
> On Monday, November 23, 2020 06:15:09 AM Sven Hartge wrote:
> > Joe <joe@jretrading.com> wrote:
> > > That's why we have IMAP, which doesn't use mbox.
> > 
> > The IMAP protocal and the backend storage have no connection.
> 
> Well, they do in a way -- if you use IMAP from your ISP for example, you don't 
> need local storage on your email device (not any of mbox, maildir, or 
> whatever).
> 
> (I don't use IMAP (I use POP3), but I assume that if I were using IMAP as 
> described, I could save emails on my local device, but I'm not clear on that 
> mechanism / storage method.)
 
It depends on the software used to retrieve the e-mails (via IMAP), e.g.  
as far as I recall getmail can be configured to use either mbox or 
Maildir as local storage.

Kind regards,
Andrei
-- 
http://wiki.debian.org/FAQsFromDebianUser

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.debian.user

csiph-web