Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #95784 > unrolled thread

Why Python is not both an interpreter and a compiler?

Back to article view | Back to comp.lang.python

Contents

  Why Python is not both an interpreter and a compiler? Mahan Marwat <mahanmarwat@gmail.com> - 2015-08-31 01:35 -0700
    Re: Why Python is not both an interpreter and a compiler? Chris Angelico <rosuav@gmail.com> - 2015-08-31 18:49 +1000
    Re: Why Python is not both an interpreter and a compiler? Cameron Simpson <cs@zip.com.au> - 2015-08-31 19:59 +1000
    Re: Why Python is not both an interpreter and a compiler? Ben Finney <ben+python@benfinney.id.au> - 2015-08-31 21:41 +1000
      Re: Why Python is not both an interpreter and a compiler? Marko Rauhamaa <marko@pacujo.net> - 2015-08-31 16:39 +0300
        Re: Why Python is not both an interpreter and a compiler? Mahan Marwat <mahanmarwat@gmail.com> - 2015-08-31 10:48 -0700
          Re: Why Python is not both an interpreter and a compiler? Emile van Sebille <emile@fenx.com> - 2015-08-31 10:56 -0700
          Re: Why Python is not both an interpreter and a compiler? Ben Finney <ben+python@benfinney.id.au> - 2015-09-01 09:15 +1000
          Re: Why Python is not both an interpreter and a compiler? Luca Menegotto <otlucaDELETE@DELETEyahoo.it> - 2015-09-01 07:45 +0200
            Re: Why Python is not both an interpreter and a compiler? Ian Kelly <ian.g.kelly@gmail.com> - 2015-09-01 11:21 -0600
          Re: Why Python is not both an interpreter and a compiler? Laura Creighton <lac@openend.se> - 2015-09-01 09:51 +0200
          Re: Why Python is not both an interpreter and a compiler? Steven D'Aprano <steve@pearwood.info> - 2015-09-02 01:20 +1000
            Re: Why Python is not both an interpreter and a compiler? Marko Rauhamaa <marko@pacujo.net> - 2015-09-01 19:20 +0300
              Re: Why Python is not both an interpreter and a compiler? Chris Angelico <rosuav@gmail.com> - 2015-09-02 03:06 +1000
              Re: Why Python is not both an interpreter and a compiler? Laura Creighton <lac@openend.se> - 2015-09-01 19:33 +0200
                Re: Why Python is not both an interpreter and a compiler? Marko Rauhamaa <marko@pacujo.net> - 2015-09-01 23:08 +0300
                  Re: Why Python is not both an interpreter and a compiler? Chris Angelico <rosuav@gmail.com> - 2015-09-02 10:50 +1000
                  Re: Why Python is not both an interpreter and a compiler? Laura Creighton <lac@openend.se> - 2015-09-02 09:09 +0200
                    Re: Why Python is not both an interpreter and a compiler? wxjmfauth@gmail.com - 2015-09-02 01:25 -0700
              Re: Why Python is not both an interpreter and a compiler? Laura Creighton <lac@openend.se> - 2015-09-01 19:52 +0200
              Re: Why Python is not both an interpreter and a compiler? Steven D'Aprano <steve@pearwood.info> - 2015-09-02 12:49 +1000
                Re: Why Python is not both an interpreter and a compiler? Marko Rauhamaa <marko@pacujo.net> - 2015-09-02 06:58 +0300
                Re: Why Python is not both an interpreter and a compiler? Jussi Piitulainen <harvesting@makes.address.invalid> - 2015-09-02 08:59 +0300
    Re: Why Python is not both an interpreter and a compiler? Michael Torrie <torriem@gmail.com> - 2015-09-01 21:46 -0600

Page 1 of 2  [1] 2  Next page →

#95784 — Why Python is not both an interpreter and a compiler?

What I know about an interpreter and a compiler is: they both convert source code to machine code and the only difference is, an interpreter convert it, line by line while compiler convert the whole source file.
Now if we compile a C source file on C compiler, it will produce a small executable file. But if we compile (freeze) a Python source file on Python interpreter (using cx_freeze), it will produce a big executable file.
Now the thing is C compiler does not embed a C compiler inside our program, while the Python interpreter (cx_freeze, pytoexe) embed Python interpreter inside our program, what is the reason?
The question is, why cx_freeze etc... embed Python interpreter inside our programs, they are unable to produce machine code like C compiler do?
Cant we program a language, which is both interpreted and compiled? The core developer cant add the compiling functionality to Python?

[toc] | [next] | [standalone]

#95785

On Mon, Aug 31, 2015 at 6:35 PM, Mahan Marwat <mahanmarwat@gmail.com> wrote:
> What I know about an interpreter and a compiler is: they both convert source code to machine code and the only difference is, an interpreter convert it, line by line while compiler convert the whole source file.
> Now if we compile a C source file on C compiler, it will produce a small executable file. But if we compile (freeze) a Python source file on Python interpreter (using cx_freeze), it will produce a big executable file.
> Now the thing is C compiler does not embed a C compiler inside our program, while the Python interpreter (cx_freeze, pytoexe) embed Python interpreter inside our program, what is the reason?
> The question is, why cx_freeze etc... embed Python interpreter inside our programs, they are unable to produce machine code like C compiler do?
> Cant we program a language, which is both interpreted and compiled? The core developer cant add the compiling functionality to Python?

Compiling is trivially easy for trivial programs, and becomes
virtually impossible for arbitrarily complex and dynamic programs. In
an extreme example, Python includes the ability to execute arbitrary
code:

>>> exec("print('Hello, world!')")
Hello, world!

You basically can't compile that without having the full capabilities
of the Python language. If you want something that makes Python into
an executable binary, you either want a wrapper (like cx_freeze and
pytoexe and so on), or else something like the PyPy project. Check it
out - it's pretty cool!

ChrisA

[toc] | [prev] | [next] | [standalone]

#95789

On 31Aug2015 01:35, Mahan Marwat <mahanmarwat@gmail.com> wrote:
>What I know about an interpreter and a compiler is: they both convert source code to machine code and the only difference is, an interpreter convert it, line by line while compiler convert the whole source file.

This is simplistic and misleading.

Compilation is the process of taking a program and converting it into an 
efficient binary form, attaching whatever else is needed (such a references to 
required libraries). Languages like C are usually compiled to machine code, and 
executed directly by hardware. Languages like Python are usually compiled to 
"byte code", effectively a machine code for a logical machine.

When hardware acts on machine code it is said to execute it. When a piece of 
software acts on byte code, it is said to "interpret" it. The main difference 
is the notional level of execution, and the corresponding level of abstraction 
in the machine code: machine code for a hardware CPU is normally very direct 
and contains instructions directly related to the hardware facilities exposed 
by the CPU (registers, memory, etc). Conversely, byte code (machine code for a 
more abstract interpreter) is more abstract: the basic elements will be higher 
level entities like strings or integers , and memory will be described by more 
abstract items like slots in call frames or names to look up in symbol tables.

>Now if we compile a C source file on C compiler, it will produce a small executable file. But if we compile (freeze) a Python source file on Python interpreter (using cx_freeze), it will produce a big executable file.

If you compile a C program with "static" linking you will get a much large 
program. Modern machines support dynamic library linking, where the needed 
libraries are attached to the executing program on the fly, largely when the 
program is started. That relies on the libraries being available in the 
filesystem when the program is invoked. By contrast a "staticly" linked program 
will get copies of the libraries installed into the executable when compiled.

>Now the thing is C compiler does not embed a C compiler inside our program, while the Python interpreter (cx_freeze, pytoexe) embed Python interpreter inside our program, what is the reason?

Chris Angelico has described why a Python interpreter is needed at runtime: 
Python can accept arbitrary Python code at runtime, so it needs to be able to 
compile it (to byte code) and interpret that byte code.

>The question is, why cx_freeze etc... embed Python interpreter inside our programs, they are unable to produce machine code like C compiler do?
>Cant we program a language, which is both interpreted and compiled? The core developer cant add the compiling functionality to Python?

You can go a long way there. There are alternatives to CPython which try to do 
this, such as Cython. A major obstacle to "complete" compilation is that Python 
is a dynamic language. Every variable may be bound to an object of any type at 
any time, so a lot of behaviour is dependent on the current binding, which may 
change.

Languages like C are staticly typed: their operations convert far more directly 
to machine operations (in fact, this is an explicit target of their design).  
Languages like Python aim for a more freindly and flexible programming syntax, 
and some of the price of that is more indirection between "byte code" and the 
underlying machine operations.

Cheers,
Cameron Simpson <cs@zip.com.au>

remember, information is not knowledge,
knowledge is not wisdom
wisdom is not truth
truth is not beauty
beauty is not love
love is not music
music is the best       - frank zappa

[toc] | [prev] | [next] | [standalone]

#95790

Mahan Marwat <mahanmarwat@gmail.com> writes:

> What I know about an interpreter and a compiler is: they both convert
> source code to machine code

Yes, Python implementations always compile the source code to machine
code. The target machine for the Python compiler, though, is a virtual
machine which then gets emulated by a Python runtime interpreter.

You can prove this by doing *only* the compile step, and later do
*only* the interpret step::

    $ python -m compileall foo.py
    Compiling foo.py ...

    $ python foo.pyc
    Hello, world!

So anything which is going to run your Python source code first needs
that source code to be compiled to machine code, and then later use an
interpreter to run that machine code.

That's true of pretty much every programming language today: To run it,
the source code is first compiled to a machine code, which is then
interpreted at run-time by some 

You might want to target some other machine code; ultimately, you might
want to the machine code of the CPU you intend to run the program. That
is a special case, though, and is by far not the only case of compiling
code.

> and the only difference is, an interpreter convert it, line by line
> while compiler convert the whole source file.

Hmm, by that definition, there are almost no language interpreters in
use today for running a program – not Python, not Ruby, not JavaScript,
not Perl – that meets your definition. They all need a compiler first to
produce machine code from the source code file.

> Now if we compile a C source file on C compiler, it will produce a
> small executable file. But if we compile (freeze) a Python source file
> on Python interpreter (using cx_freeze), it will produce a big
> executable file.

Yes, that's because the C language is low-level enough that a compiler
can target directly the host CPU's machine code. A Python program,
though, is written in a dynamic language, which is compiled to a virtual
machine code, which needs the Python virtual machine as well to
interpret that machine code.

> Now the thing is C compiler does not embed a C compiler inside our
> program, while the Python interpreter (cx_freeze, pytoexe) embed
> Python interpreter inside our program, what is the reason?

Because Python is a dynamic language. A great many facts about the
run-time state are deferred to be figured out at run-time, so can't be
decided at compile time. The resulting machine code assumes a very
powerful, dynamic machine language, which is not the same as the host
CPU's machine language.

> Cant we program a language, which is both interpreted and compiled?

I hope you see now that Python is both interpreted and compiled, by
necessity.

What you seem to want is a Python compiler that targets the host CPU;
but your target host CPU can't execute the dynamic runtime that Python
machine code needs.

In other words: The very dynamic nature that makes Python so expressive,
succinct, and efficient to write, makes it that much more distant from
the extremely low-level machine operations of the host CPU. That power
comes at the cost of a level of abstraction, in the Python virtual
machine interpreter.

-- 
 \       “Pray, v. To ask that the laws of the universe be annulled in |
  `\     behalf of a single petitioner confessedly unworthy.” —Ambrose |
_o__)                           Bierce, _The Devil's Dictionary_, 1906 |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#95791

Ben Finney <ben+python@benfinney.id.au>:

> Yes, that's because the C language is low-level enough that a compiler
> can target directly the host CPU's machine code. A Python program,
> though, is written in a dynamic language, which is compiled to a
> virtual machine code, which needs the Python virtual machine as well
> to interpret that machine code.

More generally, both C and Python programs are run with the assistance
of a run-time environment. C programs are compiled in such a way that a
single compiled file is enough to start up the program. Python programs
*could* easily be compiled the same way, but it generally hasn't been
considered all that useful.


Marko

[toc] | [prev] | [next] | [standalone]

#95793

> Python programs *could* easily be compiled the same way, but it generally  
> hasn't been considered all that useful.

If it hasn't been considered all that useful, then why the tools like cx_freeze, pytoexe are doing very hard!
And if it is really easy, then why cx_freeze, pytoexe developer are doing it in such a rubbish way instead of creating one (compiler)?

[toc] | [prev] | [next] | [standalone]

#95794

On 8/31/2015 10:48 AM, Mahan Marwat wrote:
>> Python programs *could* easily be compiled the same way, but it generally
>> hasn't been considered all that useful.
>
> If it hasn't been considered all that useful, then why the tools like cx_freeze, pytoexe are doing very hard!

People scratching their own itch.

> And if it is really easy, then why cx_freeze, pytoexe developer are doing it in such a rubbish way instead of creating one (compiler)?

You'd have to ask them.

Emile

[toc] | [prev] | [next] | [standalone]

#95801

Mahan Marwat <mahanmarwat@gmail.com> writes:

> > Python programs *could* easily be compiled the same way, but it generally  
> > hasn't been considered all that useful.
>
> If it hasn't been considered all that useful, then why the tools like
> cx_freeze, pytoexe are doing very hard!

Thay don't compile to machine code for the host CPU. That's the step
that “compiled the same way” refers to above, as not having been
considered all that useful.

What those tools do – compile to Python virtual machine code, and then
bundle that machine code with the virtual machine into a single
executable file – is quite useful.
-- 
 \        “Some people, when confronted with a problem, think ‘I know, |
  `\       I'll use regular expressions’. Now they have two problems.” |
_o__)                           —Jamie Zawinski, in alt.religion.emacs |
Ben Finney

[toc] | [prev] | [next] | [standalone]

#95816

Il 31/08/2015 19:48, Mahan Marwat ha scritto:

> If it hasn't been considered all that useful, then why
 > the tools like cx_freeze, pytoexe are doing very hard!

Well, I consider those tools useless at all!
I appreciate Python because, taken one or two precautions, I can easily 
port my code from one OS to another with no pain.
So, why should I loose this wonderful freedom?

-- 
Ciao!
Luca

[toc] | [prev] | [next] | [standalone]

#95839

On Mon, Aug 31, 2015 at 11:45 PM, Luca Menegotto
<otlucaDELETE@deleteyahoo.it> wrote:
> Il 31/08/2015 19:48, Mahan Marwat ha scritto:
>
>> If it hasn't been considered all that useful, then why
>
>> the tools like cx_freeze, pytoexe are doing very hard!
>
> Well, I consider those tools useless at all!
> I appreciate Python because, taken one or two precautions, I can easily port
> my code from one OS to another with no pain.
> So, why should I loose this wonderful freedom?

You don't. You can still take your unbundled code and port it just as
easily as before.

What is gained from those tools is the ability to easily distribute
your code to (Windows) users who aren't knowledgable or interested in
maintaining a Python installation on their system. It's something that
you don't likely use unless you have a specific need to do that,
however.

At my previous job where IT had everybody on Windows, we published our
code with batch file launchers onto the internal file server and
maintained several Python installations there to run them with, rather
than maintain them on the systems of 200+ users plus lab computers.
With that setup we didn't require cx_freeze, but we used it in some
cases for better network performance (it's typically faster to
download one large file from the network than hundreds of small
files).

[toc] | [prev] | [next] | [standalone]

#95824

If I understand what you are saying, then I think what you are
looking for is not a compiler, but docker.
see: https://www.docker.com/
in particular https://www.docker.com/whatisdocker

PyPy used this to produce portable PyPy binaries.  See:
https://github.com/squeaky-pl/portable-pypy/blob/master/README.rst

Why did we need to make portable PyPy binaries?

>From the PyPy downloads page:

    Linux binaries and common distributions

    Linux binaries are dynamically linked, as is usual, and thus might
    not be usable due to the sad story of linux binary
    compatibility. This means that Linux binaries are only usable on
    the distributions where they were created unless you're ready to hack
    your system by adding symlinks to the libraries it tries to open.

We had to make versions for debian stable and unstable, and all the
different versions of RHEL, and Centos, and Suse and .... every time
we turned around somebody made a new linux distribution, with new and
different versions of shared libraries, located in some new creative
idea of where the best place was to put such things.  And we had to make 32
bit and 64 bit versions, and versions for x86 and for ARM and the list
went on, and on, and on.  This made releases far harder than they
needed to be, and, from the user end meant that people had to work
rather hard just to figure out what version of pypy they should
download from our site.   Lots of people never could quite figure
it out, though we tried to help them on the pypy mailing list and
over irc ...

So finally, we used docker to make a portable version of pypy, which
you can use to biuld your own version of pypy from the nightly
sources, for instance.

I think that this is what you are looking for, as well.  But go read
up on docker and see if it suits.

Laura

[toc] | [prev] | [next] | [standalone]

#95834

On Tue, 1 Sep 2015 03:48 am, Mahan Marwat wrote:

>> Python programs *could* easily be compiled the same way, but it generally
>> hasn't been considered all that useful.
> 
> If it hasn't been considered all that useful, then why the tools like
> cx_freeze, pytoexe are doing very hard! And if it is really easy, then why
> cx_freeze, pytoexe developer are doing it in such a rubbish way instead of
> creating one (compiler)?

I believe that Marko is wrong. It is not so easy to compile Python to
machine language for real machines. That's why the compiler targets a
virtual machine instead.

Let's take the simple case of adding two numbers:

    x + y

With a real machine, this will probably compile down to a handful of
assembly language statements. But, x will have to be an integer of a
certain type, say, 32 bits, and y likewise will also have to be the same
size. And what happens if the addition overflows? Some machines may
overflow to a negative value, some might return the largest positive value.

Python addition does not do that. With Python, it doesn't matter whether x
and y are the same size, or different. In fact, Python integers are BigNums
which support numbers as big as you like, limited only by the amount of
memory. Or they could be floats, complex numbers, or fractions. Or x might
be a string, and y a list, and the + operator has to safely raise an
exception, not dump core, or worse.

So to compile Python into assembly language for a real CPU would be very
hard. Even simple instructions would require a large, complex chunk of
machine code. If only we could design our own machine with a CPU that
understood commands like "add two values together" without caring that the
values are compatible (real) machine types.

And that's what we do. Python runs in a virtual machine with "assembly
language" commands that are far, far more complex than real assembly for
real CPUs. Those commands, of course, eventually end up executing machine
code in the real CPU, but it usually does so by calling the Python runtime
engine.

There is another reason why frozen Python code is so large. It has to
include a full Python interpreter. The Python language includes a few
commands for compiling and interpreting Python source code:

- eval
- exec
- compile


so the Python runtime has to include the Python compiler and interpreter,
which effectively means that the Python runtime has to be *all* of Python,
or at least nearly all.

There may be projects that will compile Python down to machine code for a
real machine (perhaps Cython?) but doing so isn't easy, which is why most
compilers don't do it.




-- 
Steven

[toc] | [prev] | [next] | [standalone]

#95836

Steven D'Aprano <steve@pearwood.info>:

> I believe that Marko is wrong. It is not so easy to compile Python to
> machine language for real machines. That's why the compiler targets a
> virtual machine instead.

Somehow Guile manages it even though Scheme is at least as dynamic a
language as Python.

I never said a compiler would translate Python to (analogous) machine
language. I said you could easily turn CPython into a dynamic library
(run-time environment) and write a small bootstrapper that you package
into an executable archive with the Python code (whether .py or .pyc).
What results is a single executable that you can run analogously to any
other command.

In fact, the shebang notation turns any single .py file into such an
executable. The problem is if you break your program into modules. Java,
of course, solved a similar problem with .jar files (but still wouldn't
jump over the final hurdle of making the .jar files executable).


Marko

[toc] | [prev] | [next] | [standalone]

#95837

On Wed, Sep 2, 2015 at 2:20 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> In fact, the shebang notation turns any single .py file into such an
> executable. The problem is if you break your program into modules. Java,
> of course, solved a similar problem with .jar files (but still wouldn't
> jump over the final hurdle of making the .jar files executable).

The time machine strikes again! Python supports zip file execution,
which gives you the same benefit as a .jar file, plus you can slap on
a shebang and run the whole thing!

Anyone for some pyzza?

https://www.python.org/dev/peps/pep-0441/

ChrisA

[toc] | [prev] | [next] | [standalone]

#95841

In a message of Tue, 01 Sep 2015 19:20:51 +0300, Marko Rauhamaa writes:
>Somehow Guile manages it even though Scheme is at least as dynamic a
>language as Python.

But are Guile programs small?  I the OP made a categorisation error,
confusing correlation with causation.  (i.e. the presence of
feathers makes a animal able to fly).

Laura

[toc] | [prev] | [next] | [standalone]

#95846

Laura Creighton <lac@openend.se>:

> But are Guile programs small?

They can be tiny because libguile-2.0.so, the interpreter, is a dynamic
library and is installed on the computer. It's barely different from how
compiled C programs can be a few kilobytes in size because libc.so is
dynamic.

Emacs is a lisp interpreter. As part of its installation, emacs dumps
core, and the executable core file is installed as the editor we know
and love. That way, the most useful lisp modules are already preloaded
in the binary, executable image of the editor.

On my machine, the emacs executable is 15 megabytes in size.


Marko

[toc] | [prev] | [next] | [standalone]

#95849

On Wed, Sep 2, 2015 at 6:08 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Laura Creighton <lac@openend.se>:
>
>> But are Guile programs small?
>
> They can be tiny because libguile-2.0.so, the interpreter, is a dynamic
> library and is installed on the computer. It's barely different from how
> compiled C programs can be a few kilobytes in size because libc.so is
> dynamic.

And compiled C programs are notoriously hard to distribute. Can you
pick up a Guile binary and carry it to another computer? Do you have
to absolutely perfectly match the libguile version, architecture,
build settings, etc?

Also... how is this different from distributing .pyc files and
expecting people to have a Python interpreter?

ChrisA

[toc] | [prev] | [next] | [standalone]

#95857

In a message of Wed, 02 Sep 2015 10:50:15 +1000, Chris Angelico writes:
>And compiled C programs are notoriously hard to distribute. Can you
>pick up a Guile binary and carry it to another computer? Do you have
>to absolutely perfectly match the libguile version, architecture,
>build settings, etc?

This is the problem that docker has set out to solve.  
https://www.docker.com/whatisdocker
I think it is pretty neat stuff, myself.

Laura

[toc] | [prev] | [next] | [standalone]

#95858

cxfreeze in action:

D:\jm\Москва\exe.win32-3.4>jmtest.exe

Python crashes (very deeply)

golang in action:

D:\jm\Москва>hello3.exe
ASCII     abcde xyz
German    äöü ÄÖÜ ß
Polish    ąęźżńł
Russian   абвгдеж эюя
CJK       你好
French    œÿéà€
Misc      ሴé€㑖Ѓ⌴*

D:\jm\Москва>

- Don't argue, it's a cx_freeze issue, the issue is cleary
in the engine cx_freeze is using.

- Don't argue, it's a Windows issue, one may encounter
similar issues on a "Russian Linux".

20 years of development and this language has never
worked correctly outside the "ascii" world.

[toc] | [prev] | [next] | [standalone]

#95844

>But are Guile programs small?  I the OP made a categorisation error,
>confusing correlation with causation.  (i.e. the presence of
>feathers makes a animal able to fly).

s/I the/I think the/

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | comp.lang.python

csiph-web