Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.os.linux.misc > #11760 > unrolled thread

Script to parse a text file and extract unique email addresses

Back to article view | Back to comp.os.linux.misc

Contents

  Script to parse a text file and extract unique email addresses nomdeplume82008@googlemail.com - 2014-08-19 10:19 -0700
    Re: Script to parse a text file and extract unique email addresses Andreas Kohlbach <aug14.8.ankman@spamgourmet.com> - 2014-08-19 16:40 -0400
      Re: Script to parse a text file and extract unique email addresses The Natural Philosopher <tnp@invalid.invalid> - 2014-08-20 14:21 +0100
        Re: Script to parse a text file and extract unique email addresses nomdeplume82008@googlemail.com - 2014-08-20 08:52 -0700
          Re: Script to parse a text file and extract unique email addresses Andreas Kohlbach <aug14.8.ankman@spamgourmet.com> - 2014-08-20 18:26 -0400
            Re: Script to parse a text file and extract unique email addresses nomdeplume82008@googlemail.com - 2014-08-20 21:10 -0700
              Re: Script to parse a text file and extract unique email addresses Andreas Kohlbach <aug14.8.ankman@spamgourmet.com> - 2014-08-21 18:07 -0400
            Re: Script to parse a text file and extract unique email addresses Tim Watts <tw_usenet@dionic.net> - 2014-08-21 10:53 +0100
              Re: Script to parse a text file and extract unique email addresses Andreas Kohlbach <aug14.8.ankman@spamgourmet.com> - 2014-08-21 18:03 -0400
          Re: Script to parse a text file and extract unique email addresses Bill Marcum <bill.marcum57@yahoo.com> - 2014-08-21 03:34 -0400
            Re: Script to parse a text file and extract unique email addresses The Natural Philosopher <tnp@invalid.invalid> - 2014-08-21 12:12 +0100
            Re: Script to parse a text file and extract unique email addresses Chris Davies <chris-usenet@roaima.co.uk> - 2014-08-21 12:12 +0100
              Re: Script to parse a text file and extract unique email addresses The Natural Philosopher <tnp@invalid.invalid> - 2014-08-21 14:22 +0100
                Write it in C... (Was: Script to parse a text file and extract unique email addresses) gazelle@shell.xmission.com (Kenny McCormack) - 2014-08-21 13:42 +0000
                  Re: Write it in C... (Was: Script to parse a text file and extract unique email addresses) The Natural Philosopher <tnp@invalid.invalid> - 2014-08-21 15:23 +0100
                  Re: Write it in C... Rich <rich@example.invalid> - 2014-08-21 16:17 +0000
                    Re: Write it in C... The Natural Philosopher <tnp@invalid.invalid> - 2014-08-21 18:42 +0100
                    Re: Write it in C... William Unruh <unruh@invalid.ca> - 2014-08-21 18:38 +0000
          Re: Script to parse a text file and extract unique email addresses Loki Harfagr <l0k1@thedarkdesign.free.fr.INVALID> - 2014-09-24 13:18 +0000
            Re: Script to parse a text file and extract unique email addresses Doug Laidlaw <laidlaws@hotkey.net.au> - 2014-09-25 01:49 +1000
              Re: Script to parse a text file and extract unique email addresses Loki Harfagr <l0k1@thedarkdesign.free.fr.INVALID> - 2014-09-25 09:34 +0000
    Re: Script to parse a text file and extract unique email addresses Mladen Gogala <gogala.mladen@gmail.com> - 2014-09-29 00:17 +0000
      Re: Script to parse a text file and extract unique email addresses Rich <rich@example.invalid> - 2014-09-29 00:52 +0000
        Re: Script to parse a text file and extract unique email addresses gazelle@shell.xmission.com (Kenny McCormack) - 2014-09-29 01:13 +0000
      Re: Script to parse a text file and extract unique email addresses Tim Watts <tw_usenet@dionic.net> - 2014-09-29 07:40 +0100
      Re: Script to parse a text file and extract unique email addresses Eric Pozharski <whynot@pozharski.name> - 2014-09-29 09:52 +0300

Page 1 of 2  [1] 2  Next page →

#11760 — Script to parse a text file and extract unique email addresses

Hi all,

My scripting is a v. rusty and I am trying to write a script that will extract unique email addresses from an out Outlook file to build an address book.

In the output file, all the email addresses should preferably be in a single alphabetical column.

Any suggestions on how to do it? 

Thanks.

[toc] | [next] | [standalone]

#11762

nomdeplume82008@googlemail.com wrote on 19. August 2014:
>
> My scripting is a v. rusty and I am trying to write a script that will extract unique email addresses from an out Outlook file to build an address book.
>
> In the output file, all the email addresses should preferably be in a single alphabetical column.
>
> Any suggestions on how to do it? 

Isn't the Outlook file something proprietary? You might need to get some
plain text out of it first. Then some Perl or awk could extract the email
addresses from it.
-- 
Andreas

I wish my grass was emo. Then it would cut itself.

[toc] | [prev] | [next] | [standalone]

#11765

On 19/08/14 21:40, Andreas Kohlbach wrote:
> nomdeplume82008@googlemail.com wrote on 19. August 2014:
>>
>> My scripting is a v. rusty and I am trying to write a script that will extract unique email addresses from an out Outlook file to build an address book.
>>
>> In the output file, all the email addresses should preferably be in a single alphabetical column.
>>
>> Any suggestions on how to do it?
>
> Isn't the Outlook file something proprietary? You might need to get some
> plain text out of it first. Then some Perl or awk could extract the email
> addresses from it.
>
in general some kind of awk to select the column you want piped to sort 
-u to extract one instance of each duplicate.



-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#11766

On Wednesday, August 20, 2014 1:21:09 PM UTC, The Natural Philosopher wrote:
> On 19/08/14 21:40, Andreas Kohlbach wrote:
> 
> > wrote on 19. August 2014:
> 
> >>
> 
> >> My scripting is a v. rusty and I am trying to write a script that will extract unique email addresses from an out Outlook file to build an address book.
> 
> >>
> 
> >> In the output file, all the email addresses should preferably be in a single alphabetical column.
> 
> >>
> 
> >> Any suggestions on how to do it?
> 
> >
> 
> > Isn't the Outlook file something proprietary? You might need to get some
> 
> > plain text out of it first. Then some Perl or awk could extract the email
> 
> > addresses from it.
> 
> >
> 
> in general some kind of awk to select the column you want piped to sort 
> 
> -u to extract one instance of each duplicate.
> 
> 
> 
> 
> 
> 
> 
> -- 
> 
> Everything you read in newspapers is absolutely true, except for the 
> 
> rare story of which you happen to have first-hand knowledge. - Erwin Knoll

Going by columns won't work, because the email might be one in a CC/BCC list. 

I've checked the format of the emails, they are archived in the same manner as you would see if you read an email with its full headers. 

It would be best to get the script to extract characters around a @ sign. All of them appear to be enclosed in a <> symbols.

[toc] | [prev] | [next] | [standalone]

#11768

nomdeplume82008@googlemail.com wrote on 20. August 2014:
>
> On Wednesday, August 20, 2014 1:21:09 PM UTC, The Natural Philosopher wrote:
>> On 19/08/14 21:40, Andreas Kohlbach wrote:
>> 
>> > wrote on 19. August 2014:
>> 
>> >>
>> 
>> >> My scripting is a v. rusty and I am trying to write a script that will extract unique email addresses from an out Outlook file to build an address book.
>> 
>> >>
>> 
>> >> In the output file, all the email addresses should preferably be in a single alphabetical column.
>> 
>> >>
>> 
>> >> Any suggestions on how to do it?
>> 
>> >
>> 
>> > Isn't the Outlook file something proprietary? You might need to get some
>> 
>> > plain text out of it first. Then some Perl or awk could extract the email
>> 
>> > addresses from it.
>> 
>> >
>> 
>> in general some kind of awk to select the column you want piped to sort 
>> 
>> -u to extract one instance of each duplicate.
>> 
>> Everything you read in newspapers is absolutely true, except for the 
>> 
>> rare story of which you happen to have first-hand knowledge. - Erwin Knoll
>
> Going by columns won't work, because the email might be one in a CC/BCC list. 
>
> I've checked the format of the emails, they are archived in the same manner as you would see if you read an email with its full headers. 
>
> It would be best to get the script to extract characters around a @ sign. All of them appear to be enclosed in a <> symbols.

Try

grep -E -o "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b"|sed -e 's/^//'

to extract from the file.
-- 
Andreas

I wish my grass was emo. Then it would cut itself.

[toc] | [prev] | [next] | [standalone]

#11769

On Wednesday, August 20, 2014 10:26:18 PM UTC, Andreas Kohlbach wrote:
> wrote on 20. August 2014:
> 
> >
> 
> > On Wednesday, August 20, 2014 1:21:09 PM UTC, The Natural Philosopher wrote:
> 
> >> On 19/08/14 21:40, Andreas Kohlbach wrote:
> 
> >> 
> 
> >> > wrote on 19. August 2014:
> 
> >> 
> 
> >> >>
> 
> >> 
> 
> >> >> My scripting is a v. rusty and I am trying to write a script that will extract unique email addresses from an out Outlook file to build an address book.
> 
> >> 
> 
> >> >>
> 
> >> 
> 
> >> >> In the output file, all the email addresses should preferably be in a single alphabetical column.
> 
> >> 
> 
> >> >>
> 
> >> 
> 
> >> >> Any suggestions on how to do it?
> 
> >> 
> 
> >> >
> 
> >> 
> 
> >> > Isn't the Outlook file something proprietary? You might need to get some
> 
> >> 
> 
> >> > plain text out of it first. Then some Perl or awk could extract the email
> 
> >> 
> 
> >> > addresses from it.
> 
> >> 
> 
> >> >
> 
> >> 
> 
> >> in general some kind of awk to select the column you want piped to sort 
> 
> >> 
> 
> >> -u to extract one instance of each duplicate.
> 
> >> 
> 
> >> Everything you read in newspapers is absolutely true, except for the 
> 
> >> 
> 
> >> rare story of which you happen to have first-hand knowledge. - Erwin Knoll
> 
> >
> 
> > Going by columns won't work, because the email might be one in a CC/BCC list. 
> 
> >
> 
> > I've checked the format of the emails, they are archived in the same manner as you would see if you read an email with its full headers. 
> 
> >
> 
> > It would be best to get the script to extract characters around a @ sign. All of them appear to be enclosed in a <> symbols.
> 
> 
> 
> Try
> 
> 
> 
> grep -E -o "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b"|sed -e 's/^//'
> 
> 
> 
> to extract from the file.
> 
> -- 
> 
> Andreas
> 
> 
> 
> I wish my grass was emo. Then it would cut itself.

Wow thanks, that was amazing. 

Do you have a moment to explain how it works?

[toc] | [prev] | [next] | [standalone]

#11786

nomdeplume82008@googlemail.com wrote on 21. August 2014:
>
> On Wednesday, August 20, 2014 10:26:18 PM UTC, Andreas Kohlbach wrote:
>> wrote on 20. August 2014:
>> 
>> > It would be best to get the script to extract characters around a @ sign. All of them appear to be enclosed in a <> symbols.
>> 
>> Try
>> 
>> grep -E -o "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b"|sed -e 's/^//'
>> 
>> to extract from the file.
>
> Wow thanks, that was amazing. 
>
> Do you have a moment to explain how it works?

You feed the Linux (Unix has it too) grep command with the expression
following. Then pipe it through sed, which might not actually be
necessary.

The \b is "the edge of a word". The rest character ranges between "@" and ".".

Anyone can recommend him a good web page for beginners?
-- 
Andreas

I wish my grass was emo. Then it would cut itself.

[toc] | [prev] | [next] | [standalone]

#11771

On 20/08/14 23:26, Andreas Kohlbach wrote:
> Try
>
> grep -E -o "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b"|sed -e 's/^//'
>
> to extract from the file.
>

You need to include + and _ in the local part as well - a very common 
component.

[toc] | [prev] | [next] | [standalone]

#11785

Tim Watts wrote on 21. August 2014:
>
> On 20/08/14 23:26, Andreas Kohlbach wrote:
>> Try
>>
>> grep -E -o "\b[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z0-9.-]+\b"|sed -e 's/^//'
>>
>> to extract from the file.
>>
>
> You need to include + and _ in the local part as well - a very common
> component.

Thanks, added.
-- 
Andreas

I wish my grass was emo. Then it would cut itself.

[toc] | [prev] | [next] | [standalone]

#11770

On 08/20/2014 11:52 AM, nomdeplume82008@googlemail.com wrote:
> Going by columns won't work, because the email might be one in a CC/BCC list.
>
> I've checked the format of the emails, they are archived in the same
manner as you would see if you read an email with its full headers.
>
> It would be best to get the script to extract characters around a @
sign. All of them appear to be enclosed in a <> symbols.
>
For a CC/BCC list, you could convert every space or comma followed by a 
space to a newline.

[toc] | [prev] | [next] | [standalone]

#11773

On 21/08/14 08:34, Bill Marcum wrote:
> On 08/20/2014 11:52 AM, nomdeplume82008@googlemail.com wrote:
>> Going by columns won't work, because the email might be one in a
>> CC/BCC list.
>>
>> I've checked the format of the emails, they are archived in the same
> manner as you would see if you read an email with its full headers.
>>
>> It would be best to get the script to extract characters around a @
> sign. All of them appear to be enclosed in a <> symbols.
>>
> For a CC/BCC list, you could convert every space or comma followed by a
> space to a newline.

Then grep the file first for To: or From:


-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#11774

Bill Marcum <bill.marcum57@yahoo.com> wrote:
> For a CC/BCC list, you could convert every space or comma followed by a 
> space to a newline.

Only if that space wasn't quoted. This is a syntactically valid email
address: "chris davies"@roaima.co.uk

Chris

[toc] | [prev] | [next] | [standalone]

#11775

On 21/08/14 12:12, Chris Davies wrote:
> Bill Marcum <bill.marcum57@yahoo.com> wrote:
>> For a CC/BCC list, you could convert every space or comma followed by a
>> space to a newline.
>
> Only if that space wasn't quoted. This is a syntactically valid email
> address: "chris davies"@roaima.co.uk
>
> Chris
>

If its all that complicated I usually list all the cases and dust off 
the C compiler.

Because in general I find it takes less time to write the C than it does 
to formulate the regular expressions, as I know C but regexps remain 
after 30 years a totally illegible unmaintainable hodgepodge of 
illegible special characters.



-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#11777 — Write it in C... (Was: Script to parse a text file and extract unique email addresses)

In article <lt4rri$if6$1@news.albasani.net>,
The Natural Philosopher  <tnp@invalid.invalid> wrote:
...
>If its all that complicated I usually list all the cases and dust off 
>the C compiler.
>
>Because in general I find it takes less time to write the C than it does 
>to formulate the regular expressions, as I know C but regexps remain 
>after 30 years a totally illegible unmaintainable hodgepodge of 
>illegible special characters.

I've known people like that.

For example, they write & maintain a suite of production reports in C,
rather than use something sensible like SAS.

Or AWK.

Or Perl/whatever.

P.S.  I strongly doubt a normal person could do this in C without
re-inventing reg exps.

This is not, of course, to say that you couldn't do it...

-- 
Windows 95 n. (Win-doze): A 32 bit extension to a 16 bit user interface for
an 8 bit operating system based on a 4 bit architecture from a 2 bit company
that can't stand 1 bit of competition.

Modern day upgrade --> Windows XP Professional x64: Windows is now a 64 bit
tweak of a 32 bit extension to a 16 bit user interface for an 8 bit
operating system based on a 4 bit architecture from a 2 bit company that
can't stand 1 bit of competition.

[toc] | [prev] | [next] | [standalone]

#11778 — Re: Write it in C... (Was: Script to parse a text file and extract unique email addresses)

On 21/08/14 14:42, Kenny McCormack wrote:
> In article <lt4rri$if6$1@news.albasani.net>,
> The Natural Philosopher  <tnp@invalid.invalid> wrote:
> ...
>> If its all that complicated I usually list all the cases and dust off
>> the C compiler.
>>
>> Because in general I find it takes less time to write the C than it does
>> to formulate the regular expressions, as I know C but regexps remain
>> after 30 years a totally illegible unmaintainable hodgepodge of
>> illegible special characters.
>
> I've known people like that.
>
> For example, they write & maintain a suite of production reports in C,
> rather than use something sensible like SAS.
>
> Or AWK.
>
> Or Perl/whatever.
>
> P.S.  I strongly doubt a normal person could do this in C without
> re-inventing reg exps.
>
> This is not, of course, to say that you couldn't do it...
>
The point is you don't have to reinvent regexps. You want a filter that 
is not general purpose with an argument string, but a series of specific 
filters that are much simpler to write and maintain because they deal 
with one particular case, and one only.

Anyway, this is a linux newsgroup. By definition we are not normal people.




-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#11779 — Re: Write it in C...

Kenny McCormack <gazelle@shell.xmission.com> wrote:
> In article <lt4rri$if6$1@news.albasani.net>,
> The Natural Philosopher  <tnp@invalid.invalid> wrote:
> ...
> >If its all that complicated I usually list all the cases and dust off 
> >the C compiler.
> >

> I've known people like that.

> For example, they write & maintain a suite of production reports in
> C, rather than use something sensible like SAS.

> Or AWK.

> Or Perl/whatever.

> P.S.  I strongly doubt a normal person could do this in C without
> re-inventing reg exps.

The problem with using regexp's to validate the syntax of email
addresses is that RFC5322 format email addresses are surprisingly
complex.  This page purports to show a regex that would work:

http://blog.trojanhunter.com/2012/09/26/the-best-regex-to-validate-an-email-address/

Validating, while allowing all RFC5322 corner cases, in any language,
C, Perl, Regexp's, is not going to be a trivial task.

[toc] | [prev] | [next] | [standalone]

#11781 — Re: Write it in C...

On 21/08/14 17:17, Rich wrote:
> Kenny McCormack <gazelle@shell.xmission.com> wrote:
>> In article <lt4rri$if6$1@news.albasani.net>,
>> The Natural Philosopher  <tnp@invalid.invalid> wrote:
>> ...
>>> If its all that complicated I usually list all the cases and dust off
>>> the C compiler.
>>>
>
>> I've known people like that.
>
>> For example, they write & maintain a suite of production reports in
>> C, rather than use something sensible like SAS.
>
>> Or AWK.
>
>> Or Perl/whatever.
>
>> P.S.  I strongly doubt a normal person could do this in C without
>> re-inventing reg exps.
>
> The problem with using regexp's to validate the syntax of email
> addresses is that RFC5322 format email addresses are surprisingly
> complex.  This page purports to show a regex that would work:
>
> http://blog.trojanhunter.com/2012/09/26/the-best-regex-to-validate-an-email-address/
>
> Validating, while allowing all RFC5322 corner cases, in any language,
> C, Perl, Regexp's, is not going to be a trivial task.
>

That's why I like the multiple filter approach

Start by rejecting lines that are clearly rubbish, (that don't contain 
'@' for example)

Then look for ones that clearly don't contain valid email addresses - 
message ids and the like

Then it comes down to writing a much simpler filter that doesn't HAVE to 
caret for garbage, since that's gone already

If I knew the message format I could gave written all that in less time 
than it takes to argue.

-- 
Everything you read in newspapers is absolutely true, except for the 
rare story of which you happen to have first-hand knowledge. – Erwin Knoll

[toc] | [prev] | [next] | [standalone]

#11782 — Re: Write it in C...

On 2014-08-21, Rich <rich@example.invalid> wrote:
> Kenny McCormack <gazelle@shell.xmission.com> wrote:
>> In article <lt4rri$if6$1@news.albasani.net>,
>> The Natural Philosopher  <tnp@invalid.invalid> wrote:
>> ...
>> >If its all that complicated I usually list all the cases and dust off 
>> >the C compiler.
>> >
>
>> I've known people like that.
>
>> For example, they write & maintain a suite of production reports in
>> C, rather than use something sensible like SAS.
>
>> Or AWK.
>
>> Or Perl/whatever.
>
>> P.S.  I strongly doubt a normal person could do this in C without
>> re-inventing reg exps.
>
> The problem with using regexp's to validate the syntax of email
> addresses is that RFC5322 format email addresses are surprisingly
> complex.  This page purports to show a regex that would work:
>
> http://blog.trojanhunter.com/2012/09/26/the-best-regex-to-validate-an-email-address/
>
> Validating, while allowing all RFC5322 corner cases, in any language,
> C, Perl, Regexp's, is not going to be a trivial task.

And of course the OP does not want to validate, but find, the regular
expressions. Ie, You first have to separate out the the email address
from surrounding garbage, and then validate it. 

Thus  &*&>> ;;; %868"7jf "baby doll"@apple_cider.com "7899
and differentiate it from

This is a "baby doll"@terrible.traffic;(*& 
which is not a valid email address.
I think even that url regexp would not manage. 




>

[toc] | [prev] | [next] | [standalone]

#12156

Wed, 20 Aug 2014 08:52:07 -0700, nomdeplume82008 did cat :

> On Wednesday, August 20, 2014 1:21:09 PM UTC, The Natural Philosopher wrote:
>> On 19/08/14 21:40, Andreas Kohlbach wrote:
>> 
>> > wrote on 19. August 2014:
>> 
>> >>
>> 
>> >> My scripting is a v. rusty and I am trying to write a script that will extract unique email addresses from an out Outlook file to build an address book.
>> 
>> >>
>> 
>> >> In the output file, all the email addresses should preferably be in a single alphabetical column.
>> 
>> >>
>> 
>> >> Any suggestions on how to do it?
>> 
>> >
>> 
>> > Isn't the Outlook file something proprietary? You might need to get some
>> 
>> > plain text out of it first. Then some Perl or awk could extract the email
>> 
>> > addresses from it.
>> 
>> >
>> 
>> in general some kind of awk to select the column you want piped to sort 
>> 
>> -u to extract one instance of each duplicate.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> -- 
>> 
>> Everything you read in newspapers is absolutely true, except for the 
>> 
>> rare story of which you happen to have first-hand knowledge. - Erwin Knoll
> 
> Going by columns won't work, because the email might be one in a CC/BCC list. 
> 
> I've checked the format of the emails, they are archived in the same manner as you would see if you read an email with its full headers. 
> 
> It would be best to get the script to extract characters around a @ sign. All of them appear to be enclosed in a <> symbols.

If you're sure about that keypoint (enclosed in diamond) then the following script should be safer since it
doesn't try to fight against the monstruous regexp needed to describe the real possible email format:
$ gawk '/@/' RS='[<>]' yourfile

[toc] | [prev] | [next] | [standalone]

#12157

On 24 Sep 2014 13:18:58 GMT
Loki Harfagr <l0k1@thedarkdesign.free.fr.INVALID> wrote:

> > It would be best to get the script to extract characters around a @
> > sign. All of them appear to be enclosed in a <> symbols.  
> 
> If you're sure about that keypoint (enclosed in diamond) then the
> following script should be safer since it doesn't try to fight
> against the monstruous regexp needed to describe the real possible
> email format: $ gawk '/@/' RS='[<>]' yourfile

There are Web sites which test for a valid email address.  There may be
some ideas there.  The Javascript archives are sure to have something.

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.os.linux.misc

csiph-web