Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > muc.lists.qmail > #3223 > unrolled thread

TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send

Back to article view | Back to muc.lists.qmail

Contents

  TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Manvendra Bhangui <mbhangui@gmail.com> - 2022-05-18 20:22 +0530
    Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Charles Cazabon <search-web-for-address@pyropus.ca> - 2022-05-18 09:40 -0600
      Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send "Andy Bradford" <amb-sendok-1655608419.pihkcaidealnncpjikcf@bradfords.org> - 2022-05-19 21:13 -0600
    Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Manvendra Bhangui <mbhangui@gmail.com> - 2022-05-18 21:33 +0530
    Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Charles Cazabon <search-web-for-address@pyropus.ca> - 2022-05-18 12:57 -0600
      Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Manvendra Bhangui <mbhangui@gmail.com> - 2022-05-19 11:01 +0530
        Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send "Andy Bradford" <amb-sendok-1655610517.acaeoeagfmekmmidodop@bradfords.org> - 2022-05-19 21:48 -0600
      Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Charles Cazabon <search-web-for-address@pyropus.ca> - 2022-05-19 07:52 -0600
      Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Manvendra Bhangui <mbhangui@gmail.com> - 2022-05-19 19:59 +0530
        Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send "John Levine" <johnl@iecc.com> - 2022-05-19 13:00 -0400
      Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Charles Cazabon <search-web-for-address@pyropus.ca> - 2022-05-19 10:48 -0600
      Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Manvendra Bhangui <mbhangui@gmail.com> - 2022-05-20 09:02 +0530
    Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Erwin Hoffmann <feh@fehcom.de> - 2022-05-19 16:26 +0200
      Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Erwin Hoffmann <feh@fehcom.de> - 2022-05-19 17:34 +0200
    Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send Manvendra Bhangui <mbhangui@gmail.com> - 2022-05-19 20:51 +0530

#3223 — TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send

There have been two things that I have thought of doing differently in
qmail for a very long time.

1. Avoid scanning todo directory for new email. qmail-queue has the
   information of a new email. Can this be transmitted to qmail-send
   efficiently without the need for scanning todo directory?

2. Send multiple emails using as few connections as possible for remote
   deliveries rather a making new connection for every email to be
   delivered to remote hosts. This will be my next project and I
   think this will be challenging because too much to deal with tcp/ip
   sockets, timeouts, handling reconnection and stuff. Not sure I will
   succeed, but I will give it a try.

Eliminating lock/trigger mechanism.

In my experience I have found this to be costly in a very high injection
environment. Recently I have done lot of tests and high injection rates
always has a drastic impact on the qtime value (see zoverall script
from qmailanalog). However with the current hardwares, I haven't
encountered the Silly Qmail Syndrome which was often seen in the
hardware of the 90s.

So I thought of a way to do away with the lock/trigger mechanism and
henceavoid the need of scanning the todo directory for newly injected
mails byusing POSIX message queues in qmail-queue to communicate
with qmail-send.

To achieve this qmail-queue was re-written to use POSIX message queues and
POSIX shared memory to communicate with qmail-send. Here qmail-queue simply
communicates the inode number of the new message to qmail-send and
qmail-send classifies it as local or remote for delivery. The entire todo
scan using opendir(), readdir() is avoided. Additionally, qmail-send was
modified to write the current local and remote concurrency to shared
memory. This enabled a separate process to look at the shared memory
segment and dynamically increase the queue count based on the incoming
injection rate. Having the current concurrency also enables to look at the
queue without having to scan the queue directories (mess, local, remote
and todo). Just by setting the environment variable DYNAMIC_QUEUE to 1
enables point 1 and 2. Initially, this method gave worse performance than
the lock/trigger method, but after ironing out the issues with using IPC,
I'm getting at least 40% improvement. To measure the performance a script
using qmailanalog was written which could fire multiple processes to
inject mail using qmail-inject. I tested various distributions netqmail,
notqmail, s/qmail, indimail-mta, netqmail with exttodo patch. In this
process, I discovered that it is a huge benefit to have a separate processor
for todo directory.  netqmail with exttodo patch, s/qmail and indimail-mta
benefits a lot in managing a very low qtime. However the biggest impact on
delivery rate is from the fsync() calls made in qmail-queue.c, qmail-send.c,
qmail-local.c. In tests conducted, fsync made deliveries 8x slower. So if
you have a stable power supply and an OS like Linux (which I haven't seen
crashing for more than 10 years and counting), one can disable fsync.

Having the local and concurrency in shared memory also allows to have a
tmux script named 'qtop' which displays the local and remote concurrency
for each queue in top window and qmail-send logs in the bottom window. This
is done without using disk IO to list files in todo, local, remote subdirs. In
contrast, using qmail-qread drastically reduces injection and delivery
rates when injecting emails at a high rate.

There are few other things that this release achieves

1 dynamically Increase the number of queues based on the injection
  rate.

2 Rate limit remote deliveries. This used to be the biggest challenge for
  me when dealing with high volume delivery to sites like yahoo, aol and
  roadrunner.

3 A much smaller MTA, just one binary that would handle both local, remote
  delivery for small board computers. A downside is that it doesn't have the
  trust partitioning that having different processes to handle different jobs.

To achieve 1, a program qmonitor was written to look at the concurrency
values stored in the shared memory. When the queue reaches the
concurrency limit, a new queue is created with its own qmail-send process
to process the queue. The current number of queues is also maintained in
shared memory. This allows qmail-queue to know that a new queue is
available for use.

To achieve 2, prioq.c, qmail-send was modified and a new mta was written,
named  slowq-send. This special qmail-send looks at queue/ratelimit
directory for a file named 'domain'. Here 'domain' is the destination
domain and it contains a simple expression. e.g. 10/3600 means allow only
10 emails per hour to be delivered to this domain.

To achieve 3, a new MTA was written combining the code of qmail-send,
qmail-lspawn.c, qmail-rspawn.c. This binary known as qmta-send doesn't have
any trust partitioning. But it allows one to run just one systemd service
and one  binary to do both local and remote deliveries.

The code has been released as part of indimail-mta 3.0.0 release. It has
been in the works since Oct 2021. It has been difficult and hence I believe
that I will keep on finding bugs as I go along. But sharing this to just
let you folks know about this.

https://github.com/mbhangui/indimail-mta/releases/tag/v3.0.0

Few other things that this release has
1. envdir can use multiple directories. You just need to create
    soft links. This allows one to have a directory with global
    variables for all supervised services
2. softlimit now has a -q option to set message queue limits.
3. ucspi-tcp, qmail-smtpd, qmail-remote is openssl 3.0.0 ready
4. djb has this comment on line 248 in function comm_canwrite()
    in qmail-send.c
   /* XXX: could allow a bigger buffer; say 10 recipients */
   I tried this by having an environment variable TODO_CHUNK_SIZE
   to allow a variable buffer. This does have a positive impact on delivery
   rates at the cost of a delayed start.

All of this was benchmarked and documented here
https://github.com/mbhangui/indimail-mta/tree/master/indimail-mta-x/qmail-perf
The google sheet also has the raw data but those sheets are hidden.
Once can enable them using the unhide option in the google sheet menu.
The tests were done on an aging 2012 sony vaio laptop. But this is what I have.
Better hardware could show different results. Also FreeBSD showed
very good inject speeds for netqmail, notqmail, s/qmail but my
FreeBSD is on virtualbox and most probably the fsync() calls do not
actually do fsync(). If someone has a spare box available and
can give login access for a month, I can redo the tests on FreeBSD.

One of the few things I discovered was
1. The conf split doesn't have much impact. Increasing
     conf split gives a very marginal improvement on
     inject and delivery speeds.
2. dynamically linked qmail-queue reduces injection speeds.
     This was a big reason why indimail-mta had the worst
     injection rates. I have now made qmail-queue statically
     linked in indimail-mta.
3. zfs for the queue directory gives the best performance
    if fsync is enabled. ext4 gives the best performance if
    fsync is disabled. This is for file creation speed
4. zfs gives the worst performance for file deletion
5. file access times are more or less similar for ext4
    and zfs
6. The access times and deletion times doesn't have
     high granularity in the tests because I used the
     unix time command which gives times only
     upto two decimal places.
7. I did the tests on FreeBSD too and got good
    speeds. But my FreeBSD is on a virtualbox
    and I think the virtual box disk io is cheating on
    fsync(). One of these days I will carry out by
    installing FreeBSD natively on a separate hard
    disk.
8. Having a separate process to scan the todo drastically improves
    the qtime. It is instantly visible if you use the zoverall script from
    qmailanalog.
9. external todo processor has a remarkable impact on the
    local concurrency. The concurrency never reaches high
    values with high inject rates.
10. For a large amount of mail injection, it is almost impossible to
     beat  the speed of netqmail.

--
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [next] | [standalone]

#3224

Manvendra Bhangui <mbhangui@gmail.com> wrote:
> 
> 1. Avoid scanning todo directory for new email. qmail-queue has the
>    information of a new email. Can this be transmitted to qmail-send
>    efficiently without the need for scanning todo directory?

As you noted, I would have concerns about the weakening (shared memory etc) or
removal (combining multiple qmail programs) of the partitioning between
different parts of qmail.

ext-todo does work quite well.  Have you found that with that, on modern
hardware, that the todo scanning is still a bottleneck for the system?  I'm
not saying I doubt you, just that I haven't run into it myself lately.

Regarding combining multiple qmail programs to run on tiny embedded systems
and whatnot -- I don't think running a full MTA is a sensible choice there.
Tiny systems should be using nullmailer or ssmtp or some other equivalent to
send all their mail to a smarthost, where the real MTA lives.

> 2. Send multiple emails using as few connections as possible for remote
>    deliveries rather a making new connection for every email to be
>    delivered to remote hosts. This will be my next project and I
>    think this will be challenging because too much to deal with tcp/ip
>    sockets, timeouts, handling reconnection and stuff. Not sure I will
>    succeed, but I will give it a try.

qmail's architecture makes it difficult - you have to have qmail-send change
to supplying multiple messages and recipients to qmail-remote, which is
significant surgery.

The traditional solution to this for dealing with mail to the big hosts --
Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in a
maildir and periodically run maildirsmtp, which then delivers all accumulated
mail over a single persistent connection.  It works quite well.  The number of
huge mail providers which really benefit from this type of optimization is
reasonably small, so this is a workable solution.

Charles
-- 
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:                  http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3236

Thus said Manvendra Bhangui on Thu, 19 May 2022 19:59:02 +0530:

> I can run two maildirsmtp in parallel? Right?

Definitely. You  only want one maildir  per domain in this  case because
you aren't delivering all email to one relay. So in virtualdomains:

gmail.com:serial-gmail
yahoo.com:serial-yahoo

Then in ~serial/.qmail-gmail-default:

./maildir/gmail/

And in ~serial/.qmail-yahoo-default:

./maildir/yahoo/

And of course (as "serial" user)

maildirmake ~/maildir/gmail
maildirmake ~/maildir/yahoo

Then you just need to run (as "serial" user):

maildirsmtp ~/maildir/gmail serial-gmail- $(dnsmx gmail.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)

maildirsmtp ~/maildir/yahoo serial-yahoo- $(dnsmx yahoo.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)

The commands  for emptying  the maildirs  can be put  in a  cronjob. For
parallelism just have multiple crontab entries. You may want to make the
MX lookup more robust.

This example is adapted from TOISP in the serialmail sources.

Andy


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3225

On Wed, 18 May 2022 at 21:17, Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:
> As you noted, I would have concerns about the weakening (shared memory etc) or
> removal (combining multiple qmail programs) of the partitioning between
> different parts of qmail.

Yes. That's the concern. The trust partitioning gets eliminated by
combining the functionality of lspawn, rspawn, qmail-send in one
binary. This is the reason I run only on my raspberry pis which run
only on my internal network. They are non-critical just running music
player daemon, librespotify and airplay service.

> ext-todo does work quite well.  Have you found that with that, on modern
> hardware, that the todo scanning is still a bottleneck for the system?  I'm
> not saying I doubt you, just that I haven't run into it myself lately.

I did mention that. I have never been able to simulate silly qmail
syndrome since ages. But exttodo has a drastic improvement on the
'qtime' parameter that zoverall measures. It was visible in all my
tests. The effect of the exttodo is the mail stays in the queue for
shorter period of time compared to without the exttodo.

> Regarding combining multiple qmail programs to run on tiny embedded systems
> and whatnot -- I don't think running a full MTA is a sensible choice there.
> Tiny systems should be using nullmailer or ssmtp or some other equivalent to
> send all their mail to a smarthost, where the real MTA lives.
>
Indeed. But this was more like a fun project and nothing else. Since
I'm very familiar with the qmail code, I prefer to run this on my
raspberry PI boards that I use for my music listening. The thing is
that I have automated the whole process of updating the code on git
and have the binaries built automatically on openSUSE build service.
My raspberry pi have a script running under supervise which do apt-get
update.

> > 2. Send multiple emails using as few connections as possible for remote
> >    deliveries rather a making new connection for every email to be
> >    delivered to remote hosts. This will be my next project and I
> >    think this will be challenging because too much to deal with tcp/ip
> >    sockets, timeouts, handling reconnection and stuff. Not sure I will
> >    succeed, but I will give it a try.
>
> qmail's architecture makes it difficult - you have to have qmail-send change
> to supplying multiple messages and recipients to qmail-remote, which is
> significant surgery.

I do visualize this to be very difficult and almost impossible but
still want to give it a try just for the love of programming.

> The traditional solution to this for dealing with mail to the big hosts --
> Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in a
> maildir and periodically run maildirsmtp, which then delivers all accumulated
> mail over a single persistent connection.  It works quite well.  The number of
> huge mail providers which really benefit from this type of optimization is
> reasonably small, so this is a workable solution.

Let me think of this approach and see if I can come up with a solution
that is simple and achieves the object of rate controlled delivery
which is easily configurable by the end user.

-- 
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3226

Manvendra Bhangui <mbhangui@gmail.com> wrote:
> On Wed, 18 May 2022 at 21:17, Charles Cazabon
> 
> > Have you found that with that, on modern hardware, that the todo scanning
> > is still a bottleneck for the system?  I'm not saying I doubt you, just
> > that I haven't run into it myself lately.
> 
> I did mention that. I have never been able to simulate silly qmail syndrome
> since ages.

I wasn't specifically asking about that syndrome -- more just that, if you
have a qmail install today, with ext-todo, on reasonable modern hardware, do
you still run into issues where high injection rates result in slow deliveries
(from any cause)?

I'm trying to see if that's why you propose eliminating what you call the lock
/ trigger mechanism -- or whether you just do it "for fun".

> > The traditional solution to this for dealing with mail to the big hosts --
> > Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in
> > a maildir and periodically run maildirsmtp, which then delivers all
> > accumulated mail over a single persistent connection.
>
> Let me think of this approach and see if I can come up with a solution that
> is simple and achieves the object of rate controlled delivery which is
> easily configurable by the end user.

Do you even need to rate-limit if you've only got one connection to Gmail etc
open at a time?  maildirsmtp processes messages serially.

If you really wanted to add delays between messages sent with maildirsmtp, it
would be a very simple change to the code.  Although I would think Gmail etc
would be pretty agressive about dropping idle incoming SMTP connections, so I
doubt it's necessary or desirable.

Charles
-- 
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:                  http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3227

On Thu, 19 May 2022 at 00:34, Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:
>
> Manvendra Bhangui <mbhangui@gmail.com> wrote:
> > On Wed, 18 May 2022 at 21:17, Charles Cazabon
> >
> > > Have you found that with that, on modern hardware, that the todo scanning
> > > is still a bottleneck for the system?  I'm not saying I doubt you, just
> > > that I haven't run into it myself lately.
> >
> > I did mention that. I have never been able to simulate silly qmail syndrome
> > since ages.
>
> I wasn't specifically asking about that syndrome -- more just that, if you
> have a qmail install today, with ext-todo, on reasonable modern hardware, do
> you still run into issues where high injection rates result in slow deliveries
> (from any cause)?

Few Users are never happy with whatever speed you give them. The whole
point was to try this and also to improve the delivery rate for a few
folks who use this for marketing emails which sometimes amount to more
than 100k emails per day. I have no means to do a large number of
remote deliveries. So a benchmark was done for local deliveries using
my old laptop. The results were a bit disappointing because it gave
marginal improvement for the effort put in to write the code.

netqmail with exttodo - 31.08 emails/sec with qtime of 0.34 secs
netqmail - 33.17 emails/sec with qtime of 36.56 secs
indimail-mta lock/trigger with fsync - 40.69 emails/sec with qtime of 0.2 secs
indimail-mta ipc with fsync - 40.42 emails/sec with qtime of 0.38 secs
indimail-mta lock/trigger with fsync but delivery in chunks - 44.7
emails/sec with qtime of 9.34 secs
indimail-mta ipc with fsync but delivery in chunks - 71.77 emails/sec
with qtime of 15.44 secs
indimail-mta lock/trigger without fsync - 167.72 emails/sec with qtime
of 7.49 secs
indimail-mta ipc without fsync - 170.14 emails/sec with qtime of 3.65
secs with qtime of 3.65 secs
indimail-mta lock/trigger without fsync, but delivery in chunks -
174.6 emails/sec with qtime of 6.66 secs
iindimail-mta ipc without fsync, but delivery in chunks - 179.11
emails/sec with qtime of 7.14 secs

Regarding delivery in chunks, see djb's comment in
qmail-1.03/qmail-send.c on line 248, comm_canwrite() function. For
marketing campaigns, one of my users uses this method where qmail-send
picks up deliveries in chunks of 5000 emails. delivery doesn't start
till qmail-send has 5000 emails to schedule in one go. The del_start()
function just returns if the comm_buff doesn't have that many emails
within an elapsed period of time (configurable).

> I'm trying to see if that's why you propose eliminating what you call the lock
> / trigger mechanism -- or whether you just do it "for fun".
>

I'm not proposing it. It was something that I felt like doing and I
felt it would reduce the time because scanning of todo directory is
totally avoided. qmail-queue communicates the inode to qmail-send,
avoiding the need for qmail-send to do opendir, readdir of the todo
subdir every few seconds. I thought that it would reduce the time it
takes to finish marketing campaigns, promotions, newsletters and
stuff. In this exercise what I observed is that the fsync() calls in
qmail-queue.c, qmail-send.c, qmail-local.c have the biggest impact.
The fsync calls make qmail crash-proof at the cost of delivery rates
and increased electricity consumption. Only after the benchmarking was
I able to find out that using Posix IPC gives very marginal
improvement at the cost of system complexities managing various ipc
limits (sysctl stuff). The IPC method requires some more work to be
done on my side. I now have the ability to increase the number of
qmail-send processes based on the workload. The thing is that I have
to try this out with a higher workload and that is something pending
at my end due to lack of good hardware at home. One of my users has
promised me access to his hardware, so more tests when I get access to
his hardware.

> > > The traditional solution to this for dealing with mail to the big hosts --
> > > Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in
> > > a maildir and periodically run maildirsmtp, which then delivers all
> > > accumulated mail over a single persistent connection.
> >
> > Let me think of this approach and see if I can come up with a solution that
> > is simple and achieves the object of rate controlled delivery which is
> > easily configurable by the end user.
>
> Do you even need to rate-limit if you've only got one connection to Gmail etc
> open at a time?  maildirsmtp processes messages serially.
>
> If you really wanted to add delays between messages sent with maildirsmtp, it
> would be a very simple change to the code.  Although I would think Gmail etc
> would be pretty agressive about dropping idle incoming SMTP connections, so I
> doubt it's necessary or desirable.
>
IME, gmail does better than Yahoo. Yahoo does aggressive rate limiting
and the concurrency slots get used for mail delivery attempts which
anyway goes into deferral. So with the rate limiting code, qmail-send
just doesn't pick up mails for Yahoo, gmail, etc and uses the
available concurrency slot for delivering mails to hosts that are not
rate limiting. The rate limiting code also records for each domain the
delivery rate. When the delivery rate falls below a number, only then
qmail-send picks up the email and communicates to qmail-rspawn to make
the remote delivery attempt. The whole process of rate limiting is
done by a utility named drate which the end user uses. e.g. the
command

# drate -d yahoo.com -r 100/3600

rate limits emails to yahoo at 100 emails per hour. Having this kind
of scheme allows emails to non-big sites to be delivered without
delays. Without the rate limit code, delivery to all sites gets
delayed by delivery attempts to the big sites.

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3237

Thus said Manvendra Bhangui on Fri, 20 May 2022 09:02:03 +0530:

> Awesome thanks  Andy. That was  a very detailed  on post how  to setup
> maildirsmtp  and will  go a  long way  for anyone  who wants  to setup
> maildirsmtp.

Apparently I hadn't  yet seen Charles' very similar  set of instructions
or I may not have responded.

I  too  would actually  recommend  using  daemontools  to get  a  better
experience than  cronjob, but as he  said, the same can  be accomplished
with setlock.

Andy


--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3228

Manvendra Bhangui <mbhangui@gmail.com> wrote:
> >
> > Do you even need to rate-limit if you've only got one connection to Gmail
> > etc open at a time?  maildirsmtp processes messages serially.
> > doubt it's necessary or desirable.
>
> [...] The rate limiting code also records for each domain the
> delivery rate. When the delivery rate falls below a number, only then
> qmail-send picks up the email and communicates to qmail-rspawn to make
> the remote delivery attempt.

I would be curious to see if Yahoo rate-limits incoming mail with the
maildirsmtp method.  With only one connection open at a time, and that
connection being re-used serially (rather than establishing a new connection
for each message), it would not surprise me if their limits were much more
generous, or even not applied at all.

Charles
-- 
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:                  http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3230

On Thu, 19 May 2022 at 19:30, Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:
>
> Manvendra Bhangui <mbhangui@gmail.com> wrote:
> > [...] The rate limiting code also records for each domain the
> > delivery rate. When the delivery rate falls below a number, only then
> > qmail-send picks up the email and communicates to qmail-rspawn to make
> > the remote delivery attempt.
>
> I would be curious to see if Yahoo rate-limits incoming mail with the
> maildirsmtp method.  With only one connection open at a time, and that
> connection being re-used serially (rather than establishing a new connection
> for each message), it would not surprise me if their limits were much more
> generous, or even not applied at all.

Interesting. Yahoo could be rate limiting because of multiple
connections from the same IP, which makes use of maildirsmtp
attractive. So it may not be the number of mails sent. So what you are
saying basically is

1. have the emails to these domains delivered locally to a Maildir.
2. run maildirsmtp supplying the appropriate prefix. I guess I can run
as many maildirsmtp as the number of such domains. So if I have
yahoo.com and gmail.com, I can run two maildirsmtp in parallel? Right?

I will run this idea with my user and try this and see if this can
improve the delivery process. Thank you

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3235

It appears that Manvendra Bhangui  <mbhangui@gmail.com> said:
>1. have the emails to these domains delivered locally to a Maildir.
>2. run maildirsmtp supplying the appropriate prefix. I guess I can run
>as many maildirsmtp as the number of such domains. So if I have
>yahoo.com and gmail.com, I can run two maildirsmtp in parallel? Right?

Right, so long as they have a different prefix. I agree that you get
rate limited on parallel deliveries, not sequential ones.

Another possibility is to use multiple instances of qmail. My main instance
is in /var/qmail, a second one in /var/qmail-straw for the recipients who
want to get their mail through a straw rather than the usual firehose.

The concurrencyremote in the main instance is 30, in qmail-straw is 3.
I have virtualdomains entries in the main instance that deliver mail
to a tiny script that requeues them in qmail-straw. Works great.

By the way, I have found that Yahoo and Comcast need the straw treatment
but not Gmail.

R's,
John

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3234

Manvendra Bhangui <mbhangui@gmail.com> wrote:
> On Thu, 19 May 2022 at 19:30, Charles Cazabon wrote:
> >
> > I would be curious to see if Yahoo rate-limits incoming mail with the
> > maildirsmtp method.  With only one connection open at a time, and that
> > connection being re-used serially (rather than establishing a new connection
> > for each message), it would not surprise me if their limits were much more
> > generous, or even not applied at all.
> 
> Interesting. Yahoo could be rate limiting because of multiple
> connections from the same IP, which makes use of maildirsmtp
> attractive.

Exactly.  If you're delivering 150 messages to them with unpatched qmail,
you're opening 150 connections.  Those may not all be in parallel, but it's
likely a number of them might.  Yahoo might be doing rate-limiting primary on
"connections from <IP> per <time>" instead of/in addition to "messages from
<IP>..." or "messages from <DOMAIN>...".

> 1. have the emails to these domains delivered locally to a Maildir.
> 2. run maildirsmtp supplying the appropriate prefix.

Yes.  You set up yahoo.com or gmail.com or whatever as a virtualdomain,
probably controlled by the alias user with a suitable prefix.  Then
periodically run maildirsmtp on each of those maildirs, configured to strip
the appropriate prefix.

> I guess I can run as many maildirsmtp as the number of such domains. So if I
> have yahoo.com and gmail.com, I can run two maildirsmtp in parallel? Right?

Absolutely.  My preference is to run them as a daemontools service so that you
run them with a fixed down period between connections to that domain, and you
cdon't accidentally end up with two trying to process the same maildir at once
(a la cron), but if you choose cron you can always avoid the problem with
setlock or similar.

For anyone not quite sure what configuration I've been talking about...

.../qmail/control/virtualdomains:

  gmail.com:alias-gmail
  yahoo.com:alias-yahoo
  [...]

~alias/.qmail-gmail-default:

  ./outgoing-gmail/

/var/service/mail-gmail/run script:

  #!/bin/sh

  maildirsmtp ~alias/outgoing-gmail \
    alias-gmail- gmail-smtp-in.l.google.com myhostname.example.org

  sleep 300

> I will run this idea with my user and try this and see if this can
> improve the delivery process. Thank you

You're welcome.  Please post your results - I haven't done this comparison in
a donkey's age, and would be interested in how well it works now.

Charles

P.S. any other old hands here find their fingers typing "qmail" instead of
"gmail"?

-- 
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:                  http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3238

Awesome thanks Andy. That was a very detailed on post how to setup
maildirsmtp and will go a long way for anyone who wants to setup
maildirsmtp.

On Fri, 20 May 2022 at 08:58, Andy Bradford
<amb-sendok-1655608419.pihkcaidealnncpjikcf@bradfords.org> wrote:
>
> Thus said Manvendra Bhangui on Thu, 19 May 2022 19:59:02 +0530:
>
> > I can run two maildirsmtp in parallel? Right?
>
> Definitely. You  only want one maildir  per domain in this  case because
> you aren't delivering all email to one relay. So in virtualdomains:
>
> gmail.com:serial-gmail
> yahoo.com:serial-yahoo
>
> Then in ~serial/.qmail-gmail-default:
>
> ./maildir/gmail/
>
> And in ~serial/.qmail-yahoo-default:
>
> ./maildir/yahoo/
>
> And of course (as "serial" user)
>
> maildirmake ~/maildir/gmail
> maildirmake ~/maildir/yahoo
>
> Then you just need to run (as "serial" user):
>
> maildirsmtp ~/maildir/gmail serial-gmail- $(dnsmx gmail.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)
>
> maildirsmtp ~/maildir/yahoo serial-yahoo- $(dnsmx yahoo.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)
>
> The commands  for emptying  the maildirs  can be put  in a  cronjob. For
> parallelism just have multiple crontab entries. You may want to make the
> MX lookup more robust.
>
> This example is adapted from TOISP in the serialmail sources.
>
> Andy
>


-- 
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [next] | [standalone]

#3229

[Multipart message — attachments visible in raw view] — view raw

Hi Manvendra, 

thanks for your continuous support of qmail, your indimail project, and
your thoughts and observations sharing with us.

May I comment some of your items?

Am Mittwoch, dem 18.05.2022 um 20:22 +0530 schrieb Manvendra Bhangui:
> There have been two things that I have thought of doing differently
> in
> qmail for a very long time.
> 
> 1. Avoid scanning todo directory for new email. qmail-queue has the
>    information of a new email. Can this be transmitted to qmail-send
>    efficiently without the need for scanning todo directory?
> 

This was certainly a design decision of DJB using the file system for
both persistent storage (store&forward) and by the same token for
controlling the delivery and state. Here, we are 25 years back in time.
Given today's hardware, a revision makes sense. Further, we see
different VFSs now and of course no rotating disks any more.

It would be a challenge to provide better queuing (and faster delivery)
using the same API for eg. qmail-smtpd (and others feeding or draining
the queue). 


> 2. Send multiple emails using as few connections as possible for
> remote
>    deliveries rather a making new connection for every email to be
>    delivered to remote hosts. This will be my next project and I
>    think this will be challenging because too much to deal with
> tcp/ip
>    sockets, timeouts, handling reconnection and stuff. Not sure I
> will
>    succeed, but I will give it a try.

Same for my s/qmail; which is on my list of todos. Have a look at the
smtproutes case. 
Actually, a redesign of the queue would help in that respect. Perhaps
defining a useful control message queue protocol?


> There are few other things that this release achieves
> 
> 1 dynamically Increase the number of queues based on the injection
>   rate.

My thoughts (about s/qmail) was a bit different: Using particular
delivery channels for authenticated mails; potentially with encrypted
messages in the queue.

> 
> 2 Rate limit remote deliveries. This used to be the biggest challenge
> for
>   me when dealing with high volume delivery to sites like yahoo, aol
> and
>   roadrunner.

Hm, rate-limiting seems not to be concern for my users. 

---

Just one more question: The advantace of netqmail arises from the fact
that it minimizes the fsync() calls? 

Thanks again and also to Charles for his helpful comments.

Regards.
--eh. 

-- 
Dr. Erwin Hoffmann | www.fehcom.de
PGP key-id: 20FD6E671A94DC1E
PGP key-fingerprint:  8C6B 155B 0FDA 64F1 BCCE A6B9 20FD 6E67 1A94 DC1E

[toc] | [prev] | [next] | [standalone]

#3233

[Multipart message — attachments visible in raw view] — view raw

Hi,

Am Donnerstag, dem 19.05.2022 um 20:51 +0530 schrieb Manvendra Bhangui:
> On Thu, 19 May 2022 at 19:56, Erwin Hoffmann <feh@fehcom.de> wrote:
> > May I comment some of your items?
> > 
> Sure Erwin. You don't require my permission for that :)
> 
> ...
> 
> > > 2. Send multiple emails using as few connections as possible for
> > > remote
> > >    deliveries rather a making new connection for every email to
> > > be
> > >    delivered to remote hosts. This will be my next project and I
> > >    think this will be challenging because too much to deal with
> > > tcp/ip
> > >    sockets, timeouts, handling reconnection and stuff. Not sure I
> > > will
> > >    succeed, but I will give it a try.
> > 
> > Same for my s/qmail; which is on my list of todos. Have a look at
> > the
> > smtproutes case.
> 

in case to setup smtproutes

:mailrelay.example.com;port|user|passwd

qmail-remote (at least for s/qmail) delivers all messages to that
mailrelay in one (authenticated) SMTP transaction irrespectively of the
recipient domain: SMTP pipelining with one message (body) transmission
only.  


> I looked up your site and couldn't find details on this. Any specific
> link to this? Maybe you are talking about distributed queueing QMQ.
> This I think is similar to my concept of multiple local queues which
> could be distributed across different disks across different
> controllers, etc. How do you distribute the load across the clusters?

No, QMQ is not involved here; just plain qmail-remote + qmail-rspawn.

Regards.
--eh.

-- 
Dr. Erwin Hoffmann | www.fehcom.de
PGP key-id: 20FD6E671A94DC1E
PGP key-fingerprint:  8C6B 155B 0FDA 64F1 BCCE A6B9 20FD 6E67 1A94 DC1E

[toc] | [prev] | [next] | [standalone]

#3232

On Thu, 19 May 2022 at 19:56, Erwin Hoffmann <feh@fehcom.de> wrote:
> May I comment some of your items?
>
Sure Erwin. You don't require my permission for that :)

...

> > 2. Send multiple emails using as few connections as possible for
> > remote
> >    deliveries rather a making new connection for every email to be
> >    delivered to remote hosts. This will be my next project and I
> >    think this will be challenging because too much to deal with
> > tcp/ip
> >    sockets, timeouts, handling reconnection and stuff. Not sure I
> > will
> >    succeed, but I will give it a try.
>
> Same for my s/qmail; which is on my list of todos. Have a look at the
> smtproutes case.

I looked up your site and couldn't find details on this. Any specific
link to this? Maybe you are talking about distributed queueing QMQ.
This I think is similar to my concept of multiple local queues which
could be distributed across different disks across different
controllers, etc. How do you distribute the load across the clusters?

> Actually, a redesign of the queue would help in that respect. Perhaps
> defining a useful control message queue protocol?
...
> My thoughts (about s/qmail) was a bit different: Using particular
> delivery channels for authenticated mails; potentially with encrypted
> messages in the queue.

I look forward to some ideas on this. I did use shared memory segments
to store the concurrency. qmail-send increments this number when a
mail is added for delivery and decrements it when it gets deferred,
delivered or bounced. This has allowed me to write a script named qtop
displaying the current local and remote concurrency instantly for all
my queues
-- 
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C

--
Posted automagically by a mail2news gateway at muc.de e.V.
Please direct questions, flames, donations, etc. to news-admin@muc.de

[toc] | [prev] | [standalone]

Back to top | Article view | muc.lists.qmail

csiph-web