Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #109867 > unrolled thread

base64.b64encode(data)

Back to article view | Back to comp.lang.python

Contents

  base64.b64encode(data) Marcin Rak <mrak@sightlineinnovation.com> - 2016-06-12 11:56 -0700
    Re: base64.b64encode(data) Marko Rauhamaa <marko@pacujo.net> - 2016-06-12 22:26 +0300
    Re: base64.b64encode(data) Steven D'Aprano <steve@pearwood.info> - 2016-06-13 12:22 +1000
      Re: base64.b64encode(data) Random832 <random832@fastmail.com> - 2016-06-12 23:20 -0400
        Re: base64.b64encode(data) Steven D'Aprano <steve@pearwood.info> - 2016-06-13 15:16 +1000
          Re: base64.b64encode(data) Random832 <random832@fastmail.com> - 2016-06-13 01:33 -0400
            Re: base64.b64encode(data) Marko Rauhamaa <marko@pacujo.net> - 2016-06-13 09:45 +0300
            Re: base64.b64encode(data) Steven D'Aprano <steve@pearwood.info> - 2016-06-13 20:35 +1000
              Re: base64.b64encode(data) Random832 <random832@fastmail.com> - 2016-06-13 09:36 -0400
                Re: base64.b64encode(data) Steven D'Aprano <steve@pearwood.info> - 2016-06-22 01:56 +1000
              Re: base64.b64encode(data) Dennis Lee Bieber <wlfraed@ix.netcom.com> - 2016-06-13 20:20 -0400
          Re: base64.b64encode(data) Michael Torrie <torriem@gmail.com> - 2016-06-13 09:15 -0600
            Re: base64.b64encode(data) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2016-06-14 11:04 +1200
          Re: base64.b64encode(data) Chris Angelico <rosuav@gmail.com> - 2016-06-14 03:07 +1000
            Re: base64.b64encode(data) Gregory Ewing <greg.ewing@canterbury.ac.nz> - 2016-06-14 11:12 +1200
              Re: base64.b64encode(data) Random832 <random832@fastmail.com> - 2016-06-13 21:19 -0400

#109867 — base64.b64encode(data)

Hi to everyone.

Let's say I have some binary data, be it whatever, in the 'data' variable.  After calling the following line

b64_encoded_data = base64.b64encode(data)

my b64_encoded_data variables holds, would you believe it, a string as bytes!.
That is, the b64_encoded_data variable is of type 'bytes' and when you peek inside it's a string (made up of what seems to be only characters that exist in Base 64).  Why isn't it a string yet?  In fact, I now on that variable have to apply the decode('utf-8') method to get a string object holding the exact same sequence of characters as was held by b64_encoded_data bytes variable.

I'm a little confused as to why I would even have to apply the
.decode('utf-8') method - why doesn't base64.b64encode provide us with a result that is a 'str'?

[toc] | [next] | [standalone]

#109872

Marcin Rak <mrak@sightlineinnovation.com>:

> b64_encoded_data = base64.b64encode(data)
>
> my b64_encoded_data variables holds, would you believe it, a string as
> bytes!.

It doesn't much matter one way or another. The logic is that whenever
you encode objects, you typically want the output as bytes. However,
it's trivial to decode the bytes into a string if that's what you need.


Marko

[toc] | [prev] | [next] | [standalone]

#109879

On Mon, 13 Jun 2016 04:56 am, Marcin Rak wrote:

> Hi to everyone.
> 
> Let's say I have some binary data, be it whatever, in the 'data' variable.
>  After calling the following line
> 
> b64_encoded_data = base64.b64encode(data)
> 
> my b64_encoded_data variables holds, would you believe it, a string as
> bytes!.

That's because base64 is a bytes-to-bytes transformation. It has nothing to
do with unicode encodings.

> That is, the b64_encoded_data variable is of type 'bytes' and when you
> peek inside it's a string (made up of what seems to be only characters
> that exist in Base 64).  

If you print or otherwise display bytes, for the convenience of human
beings, those bytes are displayed as if they were ASCII. E.g. the byte 0x61
is displayed as 'a'. Good idea? Bad idea? I can see arguments either way,
but that's how it is.

Naturally after base64 encoding some bytes, you will be left with only bytes
in base64. That's the whole point of it.


> Why isn't it a string yet?

*shrug* For backwards compatibility, probably, or for historical reasons, or
because the person who write the base64 module thought that this was the
most useful behaviour.

I can promise you that had he chosen the opposite behaviour, that it returns
a str instead of bytes, there would be people complaining "why do I have to
use encode('ascii') to get bytes?".


> In fact, I now on  
> that variable have to apply the decode('utf-8') method to get a string
> object holding the exact same sequence of characters as was held by
> b64_encoded_data bytes variable.

You could also use decode('ascii'), which is probably more "correct", as the
base64 data shouldn't include anything which isn't ASCII.



-- 
Steven

[toc] | [prev] | [next] | [standalone]

#109881

On Sun, Jun 12, 2016, at 22:22, Steven D'Aprano wrote:
> That's because base64 is a bytes-to-bytes transformation. It has
> nothing to do with unicode encodings.

Nonsense. base64 is a binary-to-text encoding scheme. The output range
is specifically chosen to be safe to transmit in text protocols.

> > That is, the b64_encoded_data variable is of type 'bytes' and when
> > you peek inside it's a string (made up of what seems to be only
> > characters  that exist in Base 64).
>
> If you print or otherwise display bytes, for the convenience of human
> beings, those bytes are displayed as if they were ASCII. E.g. the byte
> 0x61 is displayed as 'a'. Good idea? Bad idea? I can see arguments
> either way, but that's how it is.

There's absolutely no rational basis for choosing "0x41-0x5A, 0x61-0x7A,
0x30-0x39, 0x2B, 0x2F" as the output range except for what characters
those values represent in ASCII. And if you needed to smuggle some
binary data through an EBCDIC system in the same manner, you would
naturally wish to convert it to the EBCDIC bytes corresponding to those
same characters.

[toc] | [prev] | [next] | [standalone]

#109887

On Mon, 13 Jun 2016 01:20 pm, Random832 wrote:

> On Sun, Jun 12, 2016, at 22:22, Steven D'Aprano wrote:
>> That's because base64 is a bytes-to-bytes transformation. It has
>> nothing to do with unicode encodings.
> 
> Nonsense. base64 is a binary-to-text encoding scheme. The output range
> is specifically chosen to be safe to transmit in text protocols.

"Safe to transmit in text protocols" surely should mean "any Unicode code
point", since all of Unicode is text. What's so special about the base64
ones?

Well, that depends on your context. For somebody who cares about sending
bits over a physical wire, their idea of "text" is not Unicode, but a
subset of ASCII *bytes*.

The end result is that after you've base64ed your "binary" data, to
get "text" data, what are you going to do with is? Treat it as Unicode code
points? Probably not. Squirt it down a wire as bytes? Almost certainly.
Looking at this from the high-level perspective of Python, that makes it
conceptually bytes not text.

Yes, I know that there's a terminology clash between communication engineers
and the programmers who work in their world, and the rest of us. We
use "text" to mean Unicode[1], they use "text" to mean "roughly 100 of the
128 bytes with the high-bit cleared, interpreted as ASCII".

But those folks are unlikely to be asking why base64 encoding a bunch of
bytes returns bytes. They *want* it to return bytes, because that's what
they're going to squirt down the wire. If you gave them Unicode, encoded
using (say) UTF-16 or UTF-32, they're likely to say "WTF are you giving me
this binary data for? Look at all these NUL bytes, what am I supposed to do
with them?!?!". (If they could cope with arbitrary bytes, they wouldn't
have base64 encoded it.) And if you gave them UTF-8, well, how would anyone
know? With base64 encoded data, it's all a subset of ASCII.

Python defines a nice clean separation between text (Unicode) and binary
data (bytes). Under that model, base64 is a transformation between
unrestricted bytes 0...255 to a restricted subset of bytes that matches
some ASCII encoded text. It shouldn't return a Unicode string, because
that's an abstract text format and we can't make any assumptions about the
implementation. Say you base64 encode some binary data:

py> base64.b64encode(b'\x01A\x11\x16')
b'AUERFg=='

Suppose instead it returned the Unicode string 'AUERFg=='. That's all well
and good, but what are you going to do with it? You can't transmit it over
a serial cable, because that almost surely is going to expect bytes, so you
have to encode it. You can't embed it in an email, because that also
expects bytes.

You could write it to a file. If the file is opened in binary mode, you have
to encode the Unicode string to bytes before you can write it. If the file
is opened in text mode, Python will accept your Unicode string and encode
it for you, which could introduce non-base64 characters into the file.
Consider if the file was opened using UTF-16:

\x00A\x00U\x00E\x00R\x00F\x00g\x00=\x00=

hardly counts as base64 in any meaningful sense.

So while I complete accept your comment about "text protocols" in the
context of the networking world, we're not in the networking world. We're
in the high-level programming language world of Python, where text does not
mean a subset of ASCII bytes, it means Unicode. And in *our* world, having
base64 return text is a mistake.






[1] Or at least we should, since the idea that only American English[2]
counts as text cannot possibly survive in the 21st Century when we're
connected to the entire world of different languages. Although I'd allow
TRON as well, if you can actually find any TRON users outside of Japan.[3]

[2] And only a subset of American English at that.

[3] Or inside Japan for that matter.

-- 
Steven

[toc] | [prev] | [next] | [standalone]

#109888

On Mon, Jun 13, 2016, at 01:16, Steven D'Aprano wrote:
> Suppose instead it returned the Unicode string 'AUERFg=='. That's all
> well and good, but what are you going to do with it? You can't
> transmit it over a serial cable, because that almost surely is going
> to expect bytes, so you have to encode it. You can't embed it in an
> email, because that also expects bytes.

Unless you're using a library that expects to receive strings and encode
them itself. Such as, in the example you so helpfully provide, a file
opened in text mode.
 
> You could write it to a file. If the file is opened in binary mode,
> you have to encode the Unicode string to bytes before you can write
> it. If the file is opened in text mode, Python will accept your
> Unicode string and encode it for you, which could introduce non-
> base64 characters into the file. Consider if the file was opened
> using UTF-16:
> 
> \x00A\x00U\x00E\x00R\x00F\x00g\x00=\x00=
> 
> hardly counts as base64 in any meaningful sense.

Why do you say these things like you assume I will agree with them. It
doesn't, in fact, introduce non-base64 characters because base64
characters are *characters*, not *bytes* and UTF-16 (or EBCDIC or
whatever) is a perfectly valid encoding of those *characters*, and the
recipient will, naturally, open that file in text mode in the same
encoding, and receive the same string, which it can then decode as
base64.

[toc] | [prev] | [next] | [standalone]

#109889

Random832 <random832@fastmail.com>:
> base64 characters are *characters*, not *bytes*

Ok, I ran into the same surprise just two days ago. But is this a big
deal?


Marko

[toc] | [prev] | [next] | [standalone]

#109893

On Mon, 13 Jun 2016 03:33 pm, Random832 wrote:

> Why do you say these things like you assume I will agree with them.


Because I gave you the benefit of the doubt that you were a reasonable
person open to good-faith discussion, rather than playing John Cleese's
role in your own personal version of the Argument Sketch :-)

I don't mind if you say "Well I'm a telecommunications engineer, and when we
talk about text protocols, this is what I mean." If I ever find myself in a
forum of telco engineers, I'll learn to use their definition too.

But this is a Python forum, and Python 3 is a language that tries very, very
hard to keep a clean separation between bytes and text, where text is
understood to mean Unicode, not a subset of ASCII-encoded bytes. Python 2
was quite happy to let the two categories bleed into each other, with
disastrous effects.

When I first started using computers, the PC world assumed that "text" meant
an ASCII-compatible subset of bytes. One character = one byte, and 'A'
meant byte 0x41 (in hex; in decimal it would be 65). Most of our wire
protocols make that same assumption, and some older file formats (like
HTML) do the same. They're remnants from a bygone age where you could get
away with calling the sequence of bytes 

48 65 6C 6C 6F 20 57 6F 72 6C 64 21

"text", because everyone[1] agreed on the same interpretation of those
bytes, namely "Hello World!". But that's no longer the case, and hasn't
been for, well to be honest it was *never* the case that 0x48 unambiguously
meant 'H', and it is certainly not the case now.

The bottom line is that critical use-cases for base64 involve transmitting
bytes, not writing arbitrary Unicode, and that's why the base64 module is
treated as a bytes to bytes transformation in Python. You can argue with me
all you like, but the docs explicitly call it this:

https://docs.python.org/3/library/codecs.html#binary-transforms

and even in Python 2 it is called "str to str", where str is understood to
be bytes-string, not Unicode:

https://docs.python.org/2/library/codecs.html#standard-encodings



And besides I've only paid for the ten minute argument.





[1] Apart from those guys using IBM mainframes. And people in foreign parts,
where they speak weird outlandish languages with bizarre characters, like
England. And users of earlier versions of ASCII, or users of variants of
ASCII that differ ever so slightly differently.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#109901

On Mon, Jun 13, 2016, at 06:35, Steven D'Aprano wrote:
> But this is a Python forum, and Python 3 is a language that tries
> very, very hard to keep a clean separation between bytes and text,

Yes, but that doesn't mean that you're right about which side of that
divide base64 output belongs on.

> where text is understood to mean Unicode, not a subset of ASCII-
> encoded bytes.

Sure. But let's not pretend that U+0020 through U+007E *aren't* unicode
characters. Base 64's output is characters. Those characters could be
encoded as ASCII, as UTF-32, as EBCDIC, and they would still be the same
characters.

At
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uuencode.html
you can see in the rationale section a specific mention of using base64
with EBCDIC, and that the characters are all invariant across all EBCDIC
encodings being part of the reason for base64 using the characters it
does (as opposed to the historical uuencode algorithm's 0x20 through
0x5F, or as opposed to using some other non-alphanumeric characters than
+ / =)

The fact that many historical standards do mix text with ASCII-encoded
bytes and treat them interchangeably, as you said, does that you have to
read carefully to see which one they mean. The problem with your
argument, though, is that in base64's case it clearly *is* text. For
example, from the original privacy-enhanced mail standards - the very
first application of base64:

RFC 989:

"1.   (Local_Form) The message text is created (e.g., via an editor)
          in the system's native character set, with lines delimited in
          accordance with local convention."

RFC 1421:

"A plaintext message
   is accepted in local form, using the host's native character set and
   line representation."

And specifically in its description of base64 ("printable encoding"):

"Proceeding from
   left to right, the bit string resulting from step 3 is encoded into
   characters which are universally representable at all sites, though
   not necessarily with the same bit patterns (e.g., although the
   character "E" is represented in an ASCII-based system as hexadecimal
   45 and as hexadecimal C5 in an EBCDIC-based system, the local
   significance of the two representations is equivalent)."

[toc] | [prev] | [next] | [standalone]

#110239

On Mon, 13 Jun 2016 11:36 pm, Random832 wrote:

> On Mon, Jun 13, 2016, at 06:35, Steven D'Aprano wrote:
>> But this is a Python forum, and Python 3 is a language that tries
>> very, very hard to keep a clean separation between bytes and text,
> 
> Yes, but that doesn't mean that you're right 

As you already know, but others might not, I asked on the Python-Dev list
why b64encode has the behaviour it has:

https://mail.python.org/pipermail/python-dev/2016-June/145166.html

**Even if** your interpretation of RFC-989 etc are correct, Python is not
bound to follow their interpretation. The RFC is a network protocol, Python
is a programming language, and our libraries can do whatever makes sense
for *programming*. And the people who migrated the Python 2 base64 lib to
Python 3 thought that it made more sense to have the functions operate on
bytes and return bytes. Other languages have made other choices:

Microsoft's base64 library in C#, C++, F# and VB takes an array of bytes as
input, and outputs a UTF-16 string:

https://msdn.microsoft.com/en-us/library/dhx0d524%28v=vs.110%29.aspx

Java's base64 encoder takes and returns bytes:

https://docs.oracle.com/javase/8/docs/api/java/util/Base64.Encoder.html

Javascript's Base64 encoder takes input as UTF-16 encoded text and returns
the same:

https://developer.mozilla.org/en-US/docs/Web/API/WindowBase64/Base64_encoding_and_decoding


RFC 989 says that their unnamed "Encode to Printable Form" uses
implementation independent characters:

  The bits resulting from the encryption operation are encoded 
  into characters which are universally representable at all 
  sites, though not necessarily with the same bit patterns (e.g., 
  although the character "E" is represented in an ASCII-based 
  system as hexadecimal 45 and as hexadecimal C5 in an EBCDIC-based
  system, the local significance of the two representations is 
  equivalent).

https://tools.ietf.org/html/rfc989


But I'm not sure how RFC 989 intends this to work in practice. If you
encrypt and encode a message on an EBCDIC machine, and the output consists
of an "E" (i.e. 0xC5, and you transmit it to an ASCII machine where you try
to decode it, it will be interpreted as an eight-bit non-ASCII character,
*not* as "E". In order for this to work, you need an additional step that
transfers byte 0xC5 (EBCDIC "E") into byte 0x45 (ASCII "E") otherwise you
get junk.

That's okay for email, since email is sent in US-ASCII[1], so any EBCDIC
machine wanting to send email must convert the header and bodies into
US-ASCII, including any Base64 attachments. But the relevance of this to
Python is pretty low.



> At
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/uuencode.html

Python's base64 module is not a re-implementation of the POSIX utility
uuencode. The uuencode utility is an application, not a library. It has its
own reasons for writing text files encoding using the local environment's
default encoding, and it explicitly states that when moving such files to
another system, they must be translated:

  [quote]
  If it was transmitted over a mail system or sent to a machine with a
  different codeset, it is assumed that, as for every other text file, 
  some translation mechanism would convert it (by the time it reached a
  user on the other system) into an appropriate codeset.
  [end quote]

In any case, the POSIX utility uuencode is free to implement whatever
high-level behaviour its authors like, just as programming language
designers are free to design their Base64 libraries to work how they like.






[1] With a few exceptions, such as binary attachments, although not all mail
servers can deal with them.


-- 
Steven

[toc] | [prev] | [next] | [standalone]

#109919

On Mon, 13 Jun 2016 09:36:06 -0400, Random832 <random832@fastmail.com>
declaimed the following:


>
>Sure. But let's not pretend that U+0020 through U+007E *aren't* unicode
>characters. Base 64's output is characters. Those characters could be
>encoded as ASCII, as UTF-32, as EBCDIC, and they would still be the same
>characters.
>
	But, as mentioned, ASCII and (to a lesser extant) EBCDIC are 1-byte per
character encodings, and are safe on 8-bit transports (ASCII on 7-bit!)
since no single byte of BASE64 would appear as any sort of control
character... The transport could include a stage punching a paper tape
which is manually moved to another system for reading.
-- 
	Wulfraed                 Dennis Lee Bieber         AF6VN
    wlfraed@ix.netcom.com    HTTP://wlfraed.home.netcom.com/

[toc] | [prev] | [next] | [standalone]

#109904

On 06/12/2016 11:16 PM, Steven D'Aprano wrote:
> "Safe to transmit in text protocols" surely should mean "any Unicode code
> point", since all of Unicode is text. What's so special about the base64
> ones?
> 
> Well, that depends on your context. For somebody who cares about sending
> bits over a physical wire, their idea of "text" is not Unicode, but a
> subset of ASCII *bytes*.

Not necessarily.  The encoding of the text containing the results of the
base64 encoding does not matter provided the letters and numbers used in
base64 can be represented.  I could take the text and paste it in an
email and send it via UTF-8, or UTF-16.  Won't make a difference
provided the decoder can deal decode that specific unicode encoding.
The other end could even cut and paste the base64 letters and numbers
out of his email body and paste it into a decoder. How the letters and
numbers got to him is immaterial and irrelevant.

Sure in the context of email base64 data is usually sent using UTF-8
encoding these days.  But there's no requirement that base64 data always
has to be encoded in ASCII, UTF-8, or LATIN1.

> The end result is that after you've base64ed your "binary" data, to
> get "text" data, what are you going to do with is? Treat it as Unicode code
> points? Probably not. 

Sure. Why not?  Write it to a text file.  Put it in an email.  Place it
in a word doc.  Print it.  Whatever.

> Squirt it down a wire as bytes? Almost certainly.

Sometimes yes.  But not always.

> Looking at this from the high-level perspective of Python, that makes it
> conceptually bytes not text.

I don't see how this is always the case.  From a high-level python
perspective it's definitely text.  That's the whole point of base64!

[toc] | [prev] | [next] | [standalone]

#109914

Michael Torrie wrote:
> On 06/12/2016 11:16 PM, Steven D'Aprano wrote:
> 
>>Squirt it down a wire as bytes? Almost certainly.
> 
> Sometimes yes.  But not always.

And even when the ultimate destination is a wire, a Python
programmer is more likely to be accessing the wire through
some high-level interface that accepts the payload to be
sent as text in the form of a Python str object.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#109905

On Tue, Jun 14, 2016 at 1:15 AM, Michael Torrie <torriem@gmail.com> wrote:
>> Looking at this from the high-level perspective of Python, that makes it
>> conceptually bytes not text.
>
> I don't see how this is always the case.  From a high-level python
> perspective it's definitely text.  That's the whole point of base64!

Maybe what Python needs is an "ascii" type that's a subclass of both
str and bytes, and requires that the contents be <0x80. It is text, so
it can be combined with text strings; but it is also bytes, so when
you combine it with bytes strings, it'll behave as most people expect.

ChrisA

[toc] | [prev] | [next] | [standalone]

#109915

Chris Angelico wrote:
> Maybe what Python needs is an "ascii" type that's a subclass of both
> str and bytes, and requires that the contents be <0x80. It is text, so
> it can be combined with text strings; but it is also bytes, so when
> you combine it with bytes strings, it'll behave as most people expect.

That would be asking for trouble, I think. It would be
letting back in a bit of the text/bytes confusion that
Python 3 worked hard to get rid of. What happens if the
bytes that you combine it with aren't in an ascii-compatible
encoding? Nothing would detect that error.

The only thing you might gain is a bit of efficiency by
removing some encoding/decoding operations. But since the
FSR, these are pretty cheap anyway when the characters are all
ascii.

They could maybe be made a bit cheaper still by arranging
some way for a bytes object and an ascii-only str object
to share underlying storage.

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#109921

On Mon, Jun 13, 2016, at 19:12, Gregory Ewing wrote:
> They could maybe be made a bit cheaper still by arranging
> some way for a bytes object and an ascii-only str object
> to share underlying storage.

While we're at it, why not allow bytes to share storage with FSR latin-1
strings and the cached UTF-8 versions of strings?

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web