Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.php > #14047 > unrolled thread

Encoding Problems

Back to article view | Back to comp.lang.php

Contents

  Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-02 15:53 +0000
    Re: Encoding Problems Jerry Stuckle <jstucklex@attglobal.net> - 2014-07-02 15:15 -0400
      Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-03 10:18 +0000
    Re: Encoding Problems Thomas 'PointedEars' Lahn <PointedEars@web.de> - 2014-07-02 21:49 +0200
      Re: Encoding Problems bill <william@TechServSys.com> - 2014-07-03 08:09 -0400
    Re: Encoding Problems Christoph Michael Becker <cmbecker69@arcor.de> - 2014-07-02 22:25 +0200
      Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-03 10:16 +0000
    Re: Encoding Problems bill <william@TechServSys.com> - 2014-07-03 08:08 -0400
      Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-03 13:17 +0000
    Re: Encoding Problems Arno Welzel <usenet@arnowelzel.de> - 2014-07-05 03:16 +0200
      Re: Encoding Problems Arno Welzel <usenet@arnowelzel.de> - 2014-07-05 03:19 +0200
        Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-10 13:57 +0000
          Re: Encoding Problems Arno Welzel <usenet@arnowelzel.de> - 2014-07-11 18:47 +0200
            Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-14 08:35 +0000
            Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-18 09:06 +0000
              Re: Encoding Problems Tim Streater <timstreater@greenbee.net> - 2014-07-18 10:29 +0100
                Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-18 09:38 +0000
                  Re: Encoding Problems Tim Streater <timstreater@greenbee.net> - 2014-07-18 11:05 +0100
                    Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-18 11:42 +0000
                      Re: Encoding Problems Tim Streater <timstreater@greenbee.net> - 2014-07-18 14:32 +0100
              Re: Encoding Problems Arno Welzel <usenet@arnowelzel.de> - 2014-07-19 09:36 +0200
                Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-29 13:58 +0000
      Re: Encoding Problems Christoph Michael Becker <cmbecker69@arcor.de> - 2014-07-05 03:28 +0200
        Re: Encoding Problems Arno Welzel <usenet@arnowelzel.de> - 2014-07-06 01:36 +0200
          Re: Encoding Problems Denis McMahon <denismfmcmahon@gmail.com> - 2014-07-06 02:16 +0000
            Re: Encoding Problems Richard Damon <Richard@Damon-Family.org> - 2014-07-05 23:19 -0400
              Re: Encoding Problems gordonb.defz8@burditt.org (Gordon Burditt) - 2014-07-06 00:21 -0500
                Re: Encoding Problems Richard Damon <Richard@Damon-Family.org> - 2014-07-06 07:14 -0400
                  Re: Encoding Problems Denis McMahon <denismfmcmahon@gmail.com> - 2014-07-07 01:24 +0000
          Re: Encoding Problems Christoph Michael Becker <cmbecker69@arcor.de> - 2014-07-06 15:30 +0200
            Re: Encoding Problems Arno Welzel <usenet@arnowelzel.de> - 2014-07-06 21:46 +0200
              Re: Encoding Problems Richard Damon <Richard@Damon-Family.org> - 2014-07-06 16:21 -0400
                Re: Encoding Problems Arno Welzel <usenet@arnowelzel.de> - 2014-07-07 03:49 +0200
      Re: Encoding Problems stef_204 <notvalid@nomail.nul> - 2014-07-10 13:18 +0000

Page 2 of 2 — ← Prev page 1 [2]

#14110

stef_204, 2014-07-18 11:06:

> On Fri, 11 Jul 2014 18:47:55 +0200, Arno Welzel wrote:
> 
>> Just keep in mind, that my example is not complete and just a suggestion
>> how to start - there should also be a case to handle text with transfer
>> encoding which does not need decoding at all and a default case to
>> handle unknown encodings.
> 
> Arno,
> 
> Sorry to ask but I am struggling with the "subject" part of the email.
> 
> I tried to find a fix but not joy, yet.
> 
> I can decode the base64 encoded subject of each email individually by 
> adding a: mb_decode_mimeheader as follows, but that's really just a 
> "hack" and not proper, IMHO.
> 
> And that only decodes $subject on the html page produced for individual 
> emails, not the top level html page/rss feed which lists all of the 
> emails.
> 
> The subjects there are still reading:
> "=?UTF-8?B?"InsertGarbledText (base64) here"=?=

imap_mime_header_decode() may help.

See <http://php.net/manual/en/function.imap-mime-header-decode.php>



-- 
Arno Welzel
http://arnowelzel.de
http://de-rec-fahrrad.de
http://fahrradzukunft.de

[toc] | [prev] | [next] | [standalone]

#14148

On Sat, 19 Jul 2014 09:36:11 +0200, Arno Welzel wrote:

> imap_mime_header_decode() may help.
> 
> See <http://php.net/manual/en/function.imap-mime-header-decode.php>
>

I'm pretty sure I tried that already but will try again. Tx.

[toc] | [prev] | [next] | [standalone]

#14066

Arno Welzel wrote:

> So extend that for the encoding:
> 
> if($msgStructure->subtype=="PLAIN")
> {
>   switch($msgStructure->encoding)
>   {
>   case 4:
>     // Body text is quoted-printable encoded
>     $body = quoted_printable_decode($body);
>     break;
> 
>   case 3:
>     // Body text is base64 encoded
>     $body = base64_decode($data);
>     break;
>   }
> 
>   $body = renderPlainText($body);
> }

What about a default clause, at least triggering a notice/warning that
the encoding is not understood?

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#14067

Christoph Michael Becker, 2014-07-05 03:28:

> Arno Welzel wrote:
> 
>> So extend that for the encoding:
>>
>> if($msgStructure->subtype=="PLAIN")
>> {
>>   switch($msgStructure->encoding)
>>   {
>>   case 4:
>>     // Body text is quoted-printable encoded
>>     $body = quoted_printable_decode($body);
>>     break;
>>
>>   case 3:
>>     // Body text is base64 encoded
>>     $body = base64_decode($data);
>>     break;
>>   }
>>
>>   $body = renderPlainText($body);
>> }
> 
> What about a default clause, at least triggering a notice/warning that
> the encoding is not understood?

Good Point. But which other encoding except no encoding at all, base64
and or quoted printable may be used?


-- 
Arno Welzel
http://arnowelzel.de
http://de-rec-fahrrad.de
http://fahrradzukunft.de

[toc] | [prev] | [next] | [standalone]

#14068

On Sun, 06 Jul 2014 01:36:20 +0200, Arno Welzel wrote:

> Christoph Michael Becker, 2014-07-05 03:28:

>> What about a default clause, at least triggering a notice/warning that
>> the encoding is not understood?

> Good Point. But which other encoding except no encoding at all, base64
> and or quoted printable may be used?

multipart/form-data

It might not be expected in an email, but then the email might be 
generated by someone looking to target some notional email application 
which had an exploitable vulnerability in that it would try and decode 
multipart/form-data in a manner that further allowed a carefully crafted 
invalid data sequence to trigger arbitrary code execution.

Now that exploit might not even be targeting the code being discussed 
here, but when script kiddie spam house sends out billions of emails in 
an attempt to exploit that vulnerability, the chances are that one of 
them will make its way into this processing chain.

Detecting and cleanly handling both unrecognised declared encoding types 
and malformed encoded data is therefore probably good practice.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#14069

On 7/5/14, 10:16 PM, Denis McMahon wrote:
> On Sun, 06 Jul 2014 01:36:20 +0200, Arno Welzel wrote:
>
>> Christoph Michael Becker, 2014-07-05 03:28:
>
>>> What about a default clause, at least triggering a notice/warning that
>>> the encoding is not understood?
>
>> Good Point. But which other encoding except no encoding at all, base64
>> and or quoted printable may be used?
>
> multipart/form-data
>
> It might not be expected in an email, but then the email might be
> generated by someone looking to target some notional email application
> which had an exploitable vulnerability in that it would try and decode
> multipart/form-data in a manner that further allowed a carefully crafted
> invalid data sequence to trigger arbitrary code execution.
>
> Now that exploit might not even be targeting the code being discussed
> here, but when script kiddie spam house sends out billions of emails in
> an attempt to exploit that vulnerability, the chances are that one of
> them will make its way into this processing chain.
>
> Detecting and cleanly handling both unrecognised declared encoding types
> and malformed encoded data is therefore probably good practice.
>

multipart/form-date would be a value for Content-Type, not a value for 
Content-Transfer-Encoding.

The defined vauls (By RFC 2045) of Content-Transfer-Encoding are:
quoted-printable
base64
binary
8bit
7bit

(Binary and 8bit are only allowed if the receiving server indicates it 
is capable of handling it).

Binary, 8bit, and 7bit imply not transform should be performed on the 
data to decode it.

[toc] | [prev] | [next] | [standalone]

#14070

> multipart/form-date would be a value for Content-Type, not a value for 
> Content-Transfer-Encoding.
> 
> The defined vauls (By RFC 2045) of Content-Transfer-Encoding are:
> quoted-printable
> base64
> binary
> 8bit
> 7bit

Never forget that viruses and malware aren't required to obey the
rules, and you could very well end up with Content-Type: spam/virus
and Content-Transfer-Encoding: rot13 .

[toc] | [prev] | [next] | [standalone]

#14071

On 7/6/14, 1:21 AM, Gordon Burditt wrote:
>> multipart/form-date would be a value for Content-Type, not a value for
>> Content-Transfer-Encoding.
>>
>> The defined vauls (By RFC 2045) of Content-Transfer-Encoding are:
>> quoted-printable
>> base64
>> binary
>> 8bit
>> 7bit
>
> Never forget that viruses and malware aren't required to obey the
> rules, and you could very well end up with Content-Type: spam/virus
> and Content-Transfer-Encoding: rot13 .
>

And if you don't properly "decode" that payload there is a problem?

Yes, you don't want you program to crash on an improper value, and 
somehow rejecting malformed messages is preferable, but ignoring the 
error isn't bad (assuming your ultimate processing of the message is 
well controlled)

[toc] | [prev] | [next] | [standalone]

#14076

On Sun, 06 Jul 2014 07:14:06 -0400, Richard Damon wrote:

> On 7/6/14, 1:21 AM, Gordon Burditt wrote:
>>> multipart/form-date would be a value for Content-Type, not a value for
>>> Content-Transfer-Encoding.

>>> The defined vauls (By RFC 2045) of Content-Transfer-Encoding are:
>>> quoted-printable base64 binary 8bit 7bit

>> Never forget that viruses and malware aren't required to obey the
>> rules, and you could very well end up with Content-Type: spam/virus and
>> Content-Transfer-Encoding: rot13 .

> And if you don't properly "decode" that payload there is a problem?

No, but doing something sensible and clean in the face of unexpected data 
values is better than just bombing out.

My original points are and remain that (a) the values you get for the 
content-transfer-encoding might not be in your list, and (b) that the 
actual content-transfer-encoding might not match the declared content-
transfer-encoding.

This combination of factors means that having a default that assumes "if 
it didn't match anything else it must be x" is a bad idea. It also means 
that it's a good idea to expect and try and cleanly detect any content 
decoding errors, on the assumption that at some point malformed content 
will arrive and you want to handle it in a manner that, at the very 
least, doesn't create a vulnerability.

-- 
Denis McMahon, denismfmcmahon@gmail.com

[toc] | [prev] | [next] | [standalone]

#14072

Arno Welzel wrote:

> Christoph Michael Becker, 2014-07-05 03:28:
> 
>> Arno Welzel wrote:
>>
>>> So extend that for the encoding:
>>>
>>> if($msgStructure->subtype=="PLAIN")
>>> {
>>>   switch($msgStructure->encoding)
>>>   {
>>>   case 4:
>>>     // Body text is quoted-printable encoded
>>>     $body = quoted_printable_decode($body);
>>>     break;
>>>
>>>   case 3:
>>>     // Body text is base64 encoded
>>>     $body = base64_decode($data);
>>>     break;
>>>   }
>>>
>>>   $body = renderPlainText($body);
>>> }
>>
>> What about a default clause, at least triggering a notice/warning that
>> the encoding is not understood?
> 
> Good Point. But which other encoding except no encoding at all, base64
> and or quoted printable may be used?

The PHP manual documents 6 values for the transfer encodings[1].
Particularly 2 (BINARY) and 5 (OTHER) seem to demand some further
handling (if only to ignore the body in these cases, what might be
necessary to avoid potential vulnerabilities).

[1] <http://www.php.net/manual/en/function.imap-fetchstructure.php>

-- 
Christoph M. Becker

[toc] | [prev] | [next] | [standalone]

#14073

Christoph Michael Becker, 2014-07-06 15:30:

> Arno Welzel wrote:
> 
>> Christoph Michael Becker, 2014-07-05 03:28:
>>
>>> Arno Welzel wrote:
>>>
>>>> So extend that for the encoding:
>>>>
>>>> if($msgStructure->subtype=="PLAIN")
>>>> {
>>>>   switch($msgStructure->encoding)
>>>>   {
>>>>   case 4:
>>>>     // Body text is quoted-printable encoded
>>>>     $body = quoted_printable_decode($body);
>>>>     break;
>>>>
>>>>   case 3:
>>>>     // Body text is base64 encoded
>>>>     $body = base64_decode($data);
>>>>     break;
>>>>   }
>>>>
>>>>   $body = renderPlainText($body);
>>>> }
>>>
>>> What about a default clause, at least triggering a notice/warning that
>>> the encoding is not understood?
>>
>> Good Point. But which other encoding except no encoding at all, base64
>> and or quoted printable may be used?
> 
> The PHP manual documents 6 values for the transfer encodings[1].
> Particularly 2 (BINARY) and 5 (OTHER) seem to demand some further
> handling (if only to ignore the body in these cases, what might be
> necessary to avoid potential vulnerabilities).
> 
> [1] <http://www.php.net/manual/en/function.imap-fetchstructure.php>

Thanks for the clarification - that's the URL I also referred to
originally ;-)



-- 
Arno Welzel
http://arnowelzel.de
http://de-rec-fahrrad.de
http://fahrradzukunft.de

[toc] | [prev] | [next] | [standalone]

#14074

On 7/6/14, 3:46 PM, Arno Welzel wrote:
> Christoph Michael Becker, 2014-07-06 15:30:
>
>> Arno Welzel wrote:
>>
>>> Christoph Michael Becker, 2014-07-05 03:28:
>>>
>>>> Arno Welzel wrote:
>>>>
>>>>> So extend that for the encoding:
>>>>>
>>>>> if($msgStructure->subtype=="PLAIN")
>>>>> {
>>>>>    switch($msgStructure->encoding)
>>>>>    {
>>>>>    case 4:
>>>>>      // Body text is quoted-printable encoded
>>>>>      $body = quoted_printable_decode($body);
>>>>>      break;
>>>>>
>>>>>    case 3:
>>>>>      // Body text is base64 encoded
>>>>>      $body = base64_decode($data);
>>>>>      break;
>>>>>    }
>>>>>
>>>>>    $body = renderPlainText($body);
>>>>> }
>>>>
>>>> What about a default clause, at least triggering a notice/warning that
>>>> the encoding is not understood?
>>>
>>> Good Point. But which other encoding except no encoding at all, base64
>>> and or quoted printable may be used?
>>
>> The PHP manual documents 6 values for the transfer encodings[1].
>> Particularly 2 (BINARY) and 5 (OTHER) seem to demand some further
>> handling (if only to ignore the body in these cases, what might be
>> necessary to avoid potential vulnerabilities).
>>
>> [1] <http://www.php.net/manual/en/function.imap-fetchstructure.php>
>
> Thanks for the clarification - that's the URL I also referred to
> originally ;-)

Looking at your original code, your base64 path is converting $data to 
$body, while the other paths are $body to $body.

The RFC defines binary as a raw encoding, meaning the message holds the 
desired byte stream. The difference between it and 7bit and 8bit is that 
in addition to using all values similar to 8bit, but also is allowed to 
have nulls(0), and CR(13) and LF(10) don't delimit lines (which no 
longer have the 998 byte length limit). If renderPlainText can't handle 
that sort of data, maybe you should discard encoding binary, but then 
just because the message doesn't say it is binary, doesn't force it to 
obey (unless your MTA checks and enforces this), so renderPlainText 
should do something "valid" for these cases anyway (even if it is just 
outputting nothing).

Similarly, "Other" probably means that the encoding wasn't validly 
specified, so you might want to reject, but you don't need to (as you 
should be able to handle in some manner what ever "garbage" is sent to 
you, even if it be rejecting or outputting nothing).

[toc] | [prev] | [next] | [standalone]

#14078

Richard Damon, 2014-07-06 22:21:

> On 7/6/14, 3:46 PM, Arno Welzel wrote:
>> Christoph Michael Becker, 2014-07-06 15:30:
>>
>>> Arno Welzel wrote:
>>>
>>>> Christoph Michael Becker, 2014-07-05 03:28:
>>>>
>>>>> Arno Welzel wrote:
>>>>>
>>>>>> So extend that for the encoding:
>>>>>>
>>>>>> if($msgStructure->subtype=="PLAIN")
>>>>>> {
>>>>>>    switch($msgStructure->encoding)
>>>>>>    {
>>>>>>    case 4:
>>>>>>      // Body text is quoted-printable encoded
>>>>>>      $body = quoted_printable_decode($body);
>>>>>>      break;
>>>>>>
>>>>>>    case 3:
>>>>>>      // Body text is base64 encoded
>>>>>>      $body = base64_decode($data);
>>>>>>      break;
>>>>>>    }
>>>>>>
>>>>>>    $body = renderPlainText($body);
>>>>>> }
>>>>>
>>>>> What about a default clause, at least triggering a notice/warning that
>>>>> the encoding is not understood?
>>>>
>>>> Good Point. But which other encoding except no encoding at all, base64
>>>> and or quoted printable may be used?
>>>
>>> The PHP manual documents 6 values for the transfer encodings[1].
>>> Particularly 2 (BINARY) and 5 (OTHER) seem to demand some further
>>> handling (if only to ignore the body in these cases, what might be
>>> necessary to avoid potential vulnerabilities).
>>>
>>> [1] <http://www.php.net/manual/en/function.imap-fetchstructure.php>
>>
>> Thanks for the clarification - that's the URL I also referred to
>> originally ;-)
> 
> Looking at your original code, your base64 path is converting $data to 
> $body, while the other paths are $body to $body.

Yep - that's why I corrected this fault in my follow-up in
<53B7529D.9050002@arnowelzel.de>.

This was just meant as suggestion how one could handle the diferent
content-transfer-encodings, not tested code ready to use. Therefore I
also mentioned the PHP manual for further reading.

Of course one should add a default case to handle "unknown"
content-transfer-encodings.


-- 
Arno Welzel
http://arnowelzel.de
http://de-rec-fahrrad.de
http://fahrradzukunft.de

[toc] | [prev] | [next] | [standalone]

#14088

On Sat, 05 Jul 2014 03:16:59 +0200, Arno Welzel wrote:

> Well - unfortunately there is no easy "put it in there and it works".
> 
> First of all: Just testing for the subtype "PLAIN" (is not enough. You
> also have to check for the encoding.
> 
> As far i see, this should be added here:
> 
> if($msgStructure->subtype=="PLAIN")
> $body = renderPlainText($body);
> 
> So extend that for the encoding:
> 
> if($msgStructure->subtype=="PLAIN")
> {
>   switch($msgStructure->encoding)
>   {
>   case 4:
>     // Body text is quoted-printable encoded $body =
>     quoted_printable_decode($body); break;
> 
>   case 3:
>     // Body text is base64 encoded $body = base64_decode($data);
>     break;
>   }
> 
>   $body = renderPlainText($body);
> }
> 
> Also see <http://php.net/manual/en/function.imap-fetchstructure.php> and
> the comments there.

Hi,
Sorry, had not seen your feedback yet.
I tried something similar but no joy--but I probably didn't do it right.
Let me try again, using your suggestion.
Tx.

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.php

csiph-web