Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #101963 > unrolled thread

importing: what does "from" do?

Back to article view | Back to comp.lang.python

Contents

  importing: what does "from" do? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-21 14:20 +0000
    Re: importing: what does "from" do? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-21 07:52 -0700
      Re: importing: what does "from" do? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-21 15:12 +0000
        Re: importing: what does "from" do? Ian Kelly <ian.g.kelly@gmail.com> - 2016-01-21 08:59 -0700
          Re: importing: what does "from" do? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-21 16:30 +0000
            Re: importing: what does "from" do? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-21 16:41 +0000
        Re: importing: what does "from" do? Steven D'Aprano <steve@pearwood.info> - 2016-01-22 03:22 +1100
    Re: importing: what does "from" do? John Gordon <gordon@panix.com> - 2016-01-21 15:31 +0000
      Re: importing: what does "from" do? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-21 15:37 +0000
        Re: importing: what does "from" do? John Gordon <gordon@panix.com> - 2016-01-21 15:44 +0000
          Re: importing: what does "from" do? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-21 15:46 +0000
            Re: importing: what does "from" do? John Gordon <gordon@panix.com> - 2016-01-21 15:59 +0000
          Re: importing: what does "from" do? "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-21 15:53 +0000
            Re: importing: what does "from" do? Steven D'Aprano <steve@pearwood.info> - 2016-01-22 03:28 +1100
    Re: importing: what does "from" do? Rustom Mody <rustompmody@gmail.com> - 2016-01-21 23:03 -0800
      Refactoring in a large code base (was: importing: what does "from" do?) Ben Finney <ben+python@benfinney.id.au> - 2016-01-22 18:54 +1100
        Re: Refactoring in a large code base Marko Rauhamaa <marko@pacujo.net> - 2016-01-22 10:29 +0200
          Re: Refactoring in a large code base Rustom Mody <rustompmody@gmail.com> - 2016-01-22 01:21 -0800
            Re: Refactoring in a large code base Marko Rauhamaa <marko@pacujo.net> - 2016-01-22 12:19 +0200
              Re: Refactoring in a large code base Chris Angelico <rosuav@gmail.com> - 2016-01-22 22:19 +1100
                Re: Refactoring in a large code base Marko Rauhamaa <marko@pacujo.net> - 2016-01-22 13:54 +0200
                  Re: Refactoring in a large code base Chris Angelico <rosuav@gmail.com> - 2016-01-22 23:28 +1100
                    Re: Refactoring in a large code base Marko Rauhamaa <marko@pacujo.net> - 2016-01-22 15:00 +0200
                      Re: Refactoring in a large code base Chris Angelico <rosuav@gmail.com> - 2016-01-23 00:26 +1100
                        Re: Refactoring in a large code base Marko Rauhamaa <marko@pacujo.net> - 2016-01-22 15:48 +0200
                          Re: Refactoring in a large code base Chris Angelico <rosuav@gmail.com> - 2016-01-23 01:09 +1100
                Re: Refactoring in a large code base Rustom Mody <rustompmody@gmail.com> - 2016-01-22 04:04 -0800
                  Re: Refactoring in a large code base Marko Rauhamaa <marko@pacujo.net> - 2016-01-22 14:21 +0200
                  Re: Refactoring in a large code base Thomas Mellman <tmellman@web.de> - 2016-01-22 12:27 +0000
                  Re: Refactoring in a large code base Chris Angelico <rosuav@gmail.com> - 2016-01-22 23:34 +1100
                    Re: Refactoring in a large code base Rustom Mody <rustompmody@gmail.com> - 2016-01-22 05:30 -0800
                      Re: Refactoring in a large code base Marko Rauhamaa <marko@pacujo.net> - 2016-01-22 15:43 +0200
                        Re: Refactoring in a large code base Rustom Mody <rustompmody@gmail.com> - 2016-01-22 05:58 -0800
                      Re: Refactoring in a large code base Chris Angelico <rosuav@gmail.com> - 2016-01-23 01:08 +1100
              Re: Refactoring in a large code base "Charles T. Smith" <cts.private.yahoo@gmail.com> - 2016-01-22 11:45 +0000

Page 2 of 2 — ← Prev page 1 [2]

#102003 — Re: Refactoring in a large code base

Chris Angelico <rosuav@gmail.com>:

> On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> So what do you do with a huge program?

Modularize. Treat each module as a separate product with its own release
cycle, documentation, apis, ownership etc.

What is a reasonable size of a module? It is something you would
consider replacing with a new implementation with a moderate effort
(say, in a single quarter).

> CPython is a large and complex program. How do you propose doing it
> "right"?

I don't know CPython specifically to give solid recommendations, but I
would imagine the core language engine should be in a repository
separate from the standard library, and most standard library modules
should be in their respective repositories and have their individual
internal release cycles.

A CPython release would then weave the package together from the
components that were previously (internally) released.


Marko

[toc] | [prev] | [next] | [standalone]

#102006 — Re: Refactoring in a large code base

On Fri, Jan 22, 2016 at 10:54 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa <marko@pacujo.net> wrote:
>> So what do you do with a huge program?
>
> Modularize. Treat each module as a separate product with its own release
> cycle, documentation, apis, ownership etc.
>
> What is a reasonable size of a module? It is something you would
> consider replacing with a new implementation with a moderate effort
> (say, in a single quarter).
>
>> CPython is a large and complex program. How do you propose doing it
>> "right"?
>
> I don't know CPython specifically to give solid recommendations, but I
> would imagine the core language engine should be in a repository
> separate from the standard library, and most standard library modules
> should be in their respective repositories and have their individual
> internal release cycles.
>
> A CPython release would then weave the package together from the
> components that were previously (internally) released.

Okay. So let's suppose we strip out huge slabs of the standard library
and make an absolutely minimal "base library", with an "extended
library" that can run on its own separate release cycle. (This has
already been discussed; the biggest problems with the idea aren't
technical, but logistical - not just for the Python devs but for
everyone who has to get approval for software upgrades.) Let's suppose
the base library consists of just the modules necessary for a basic
invocation:

rosuav@sikorsky:~$ python3 -c 'import sys; print(sorted(sys.modules.keys()))'
['__main__', '_codecs', '_collections_abc', '_frozen_importlib',
'_frozen_importlib_external', '_imp', '_io', '_signal',
'_sitebuiltins', '_stat', '_sysconfigdata', '_thread', '_warnings',
'_weakref', '_weakrefset', 'abc', 'builtins', 'codecs', 'encodings',
'encodings.aliases', 'encodings.latin_1', 'encodings.utf_8', 'errno',
'genericpath', 'io', 'marshal', 'os', 'os.path', 'posix', 'posixpath',
'site', 'stat', 'sys', 'sysconfig', 'zipimport']

Alright. Can you rewrite all of those modules in three months? Not
even digging into the language itself, just the base library. This is
the bare minimum to get a viable Python execution environment going
(you might be able to cut it down a bit, but not much), so it can't be
modularized into separate projects.

And then there's the language itself. The cpython/Python directory has
58 .c files, many of which are closely tied to each other. The
cpython/Objects directory has another 39, representing specific object
types (bytes, tuple, range, method) that are implemented in C. And
cpython/Parser has 17 more just to handle the language parser. Edits
often affect multiple files and must be kept in sync. How would you
modularize that out? Which part would you spin off as a separate
project with its own release cycle? The garbage collector? The string
object? The peephole optimizer? The import machinery? Each of these is
already too big to rewrite in three months, plus they're fairly
tightly linked to all the other modules. All that code represents the
accumulation of hundreds of thousands of fixes to prevent tens of
millions of bugs (some of which will be visible on bugs.python.org,
but most would have been found and prevented during early testing);
throwing the code away means throwing all that away.

http://www.joelonsoftware.com/articles/fog0000000069.html

I don't agree with everything Joel says, but seriously, do not waste
your time with a full rewrite - even in theory. And I can say this
from hard experience on both sides. I have an active project for a MUD
server, which was originally deployed as a byte-oriented service (it
took ASCII-compatible bytes from clients and sent those same octets
out to other clients). When I decided that the server should work with
Unicode text internally (expecting and transmitting UTF-8), I kept on
coming across stupid problems where the code had been written with
faulty assumptions, and I had to keep on fixing those. Would it have
been better to throw the code away and start over? Well, let me tell
you, it would certainly have made the Unicode handling a lot easier,
so if you're looking at starting your own project, make sure you learn
from my hassles and bake in Unicode support from the start! But that
would have meant throwing away all the bugfixes for all the bugs that
I'd noticed across the years, such as:

1) On login, typing "quit" when prompted for a user name or password
would log you out. The "passwd" (change password) command had to also
prevent you from *setting* your password to "quit", because that would
effectively lock your account against login.

2) Some clients send backspace as 08; others send FF; some send 08 20
08. Cope with them all.

3) If a bug prevents the admin account from working, there needs to be
a way to diagnose and fix that code using shell access to the back-end
server, without needing the actual admin account.

Etcetera, etcetera, etcetera. There's no way to "rewrite but keep all
the bug fixes". That's called not rewriting.

So how can you rewrite *any* large project in three months?

ChrisA

[toc] | [prev] | [next] | [standalone]

#102009 — Re: Refactoring in a large code base

Chris Angelico <rosuav@gmail.com>:

> Alright. Can you rewrite all of those modules in three months?

The point is not to rewrite modules except as a fallback for a
hopelessly badly written module.

> And then there's the language itself. The cpython/Python directory has
> 58 .c files, many of which are closely tied to each other.

Putting modularization in place after the spaghetti has been created is
very hard. It can be done, though, and good things come out of the
effort.

> I don't agree with everything Joel says, but seriously, do not waste
> your time with a full rewrite - even in theory. And I can say this
> from hard experience on both sides.

As long as you are happy with your code base, you don't have to change
everything just for an abstract principle.

However, as a matter of rule, older code bases have been bloated till
they can barely be maintained. That's when the management starts to
listen to new ideas. Better late than never.

> Would it have been better to throw the code away and start over?

Again, modularization doesn't entail rewriting -- it simply makes
localized rewriting a practical option for desperate situations.

> So how can you rewrite *any* large project in three months?

Let go of the rewriting already.


Marko

[toc] | [prev] | [next] | [standalone]

#102010 — Re: Refactoring in a large code base

On Sat, Jan 23, 2016 at 12:00 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> However, as a matter of rule, older code bases have been bloated till
> they can barely be maintained. That's when the management starts to
> listen to new ideas. Better late than never.

Okay. Start persuading "management" (presumably the PSU) that CPython
needs to be more modular, with different release cycles for different
components. Your first step is to figure out the boundaries between
those components. Get started.

ChrisA

[toc] | [prev] | [next] | [standalone]

#102013 — Re: Refactoring in a large code base

Chris Angelico <rosuav@gmail.com>:

> Okay. Start persuading "management" (presumably the PSU) that CPython
> needs to be more modular, with different release cycles for different
> components. Your first step is to figure out the boundaries between
> those components. Get started.

Gladly, I don't need to do anything about CPython. This particular
strawman was erected by you:

> CPython is a large and complex program. How do you propose doing it
> "right"?

It's up to you to take my proposal or ignore it.

However, I have had my share of windmills to battle; I'm happy to say
I've managed to bring down a few of them!


Marko

[toc] | [prev] | [next] | [standalone]

#102017 — Re: Refactoring in a large code base

On Sat, Jan 23, 2016 at 12:48 AM, Marko Rauhamaa <marko@pacujo.net> wrote:
> Chris Angelico <rosuav@gmail.com>:
>
>> Okay. Start persuading "management" (presumably the PSU) that CPython
>> needs to be more modular, with different release cycles for different
>> components. Your first step is to figure out the boundaries between
>> those components. Get started.
>
> Gladly, I don't need to do anything about CPython. This particular
> strawman was erected by you:
>
>> CPython is a large and complex program. How do you propose doing it
>> "right"?
>
> It's up to you to take my proposal or ignore it.
>
> However, I have had my share of windmills to battle; I'm happy to say
> I've managed to bring down a few of them!

Okay. You have fun releasing nothing that you aren't confident can fit
within your definitions (and by the way, you were the one who brought
up the three-month rewrite, not me); I'm going to keep on following
the principle that "practicality beats purity", and release actual
working code.

ChrisA

[toc] | [prev] | [next] | [standalone]

#102004 — Re: Refactoring in a large code base

On Friday, January 22, 2016 at 4:49:19 PM UTC+5:30, Chris Angelico wrote:
> On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa  wrote:
> > The knowhow, vision and skill is apparently very rare. On the product
> > management side, we have the famous case of Steve Jobs, who simply told
> > the engineers to go back to the drawing boards when he didn't like the
> > user experience. Most others would have simply surrendered to the
> > mediocre designs and shipped the product.
> >
> > We need similar code sanity management. Developers are given much too
> > much power to mess up the source code. That's why "legacy" is considered
> > a four-letter word among developers.
> 
> So what do you do with a huge program? Do you send it back to the
> developers and say "Do this is less lines of code"?
> 
> CPython is a large and complex program. How do you propose doing it "right"?

Put thus 'generistically' this is a rhetorical question and makes Marko look like
he's making a really foolish point

Specifically, what little Ive seen under the CPython hood looked distinctly improvable. egs.

1. My suggestion to have the docs re. generator-function vs generator-objects
cleaned up had no takers
2. My students trying to work inside the lexer made a mess because the extant lexer is a mess.
I.e. while python(3) *claims* to accept Unicode input, the actual lexer is
an ASCII lexer special-cased for unicode rather than pre-lexing utf8 to unicode

These are just specific examples that I am familiar with
Chris' general point still stands, viz take the large and complex program that is cpython
and clean up these messinesses: You will still have a large and complex program

[toc] | [prev] | [next] | [standalone]

#102005 — Re: Refactoring in a large code base

Rustom Mody <rustompmody@gmail.com>:

> These are just specific examples that I am familiar with Chris'
> general point still stands, viz take the large and complex program
> that is cpython and clean up these messinesses: You will still have a
> large and complex program

No, as long as the ugly parts are compartmentalized, you have a better
chance at refactoring them -- or replacing them altogether.

Modularization is an obvious, but under-practiced, method of managing
complexity.


Marko

[toc] | [prev] | [next] | [standalone]

#102007 — Re: Refactoring in a large code base

On Fri, 22 Jan 2016 04:04:44 -0800, Rustom Mody wrote:

> These are just specific examples that I am familiar with Chris' general
> point still stands, viz take the large and complex program that is
> cpython and clean up these messinesses: You will still have a large and
> complex program

I'm not really sure what the point is we're working on...  let me
propose these:

- unix principle is good: keep things simple, limited in scope.
  Then leverage that.

- there will always be complexity, but if the complexity is
  modularized, it's controlled.

  In particular, the complexity of a program should represent the
  complexity of the problem.  I call that "structural complexity".
  To be avoided, corrected, is "superficial complexity",
  where the complexity of a system is squished into a single (or
  reduced number of) planes.  Like vomiting a program onto a desk.

- "Advice" that the program needs to be refracted is generally not helpful.

[toc] | [prev] | [next] | [standalone]

#102008 — Re: Refactoring in a large code base

On Fri, Jan 22, 2016 at 11:04 PM, Rustom Mody <rustompmody@gmail.com> wrote:
> On Friday, January 22, 2016 at 4:49:19 PM UTC+5:30, Chris Angelico wrote:
>> On Fri, Jan 22, 2016 at 9:19 PM, Marko Rauhamaa  wrote:
>> > The knowhow, vision and skill is apparently very rare. On the product
>> > management side, we have the famous case of Steve Jobs, who simply told
>> > the engineers to go back to the drawing boards when he didn't like the
>> > user experience. Most others would have simply surrendered to the
>> > mediocre designs and shipped the product.
>> >
>> > We need similar code sanity management. Developers are given much too
>> > much power to mess up the source code. That's why "legacy" is considered
>> > a four-letter word among developers.
>>
>> So what do you do with a huge program? Do you send it back to the
>> developers and say "Do this is less lines of code"?
>>
>> CPython is a large and complex program. How do you propose doing it "right"?
>
> Put thus 'generistically' this is a rhetorical question and makes Marko look like
> he's making a really foolish point
>
> Specifically, what little Ive seen under the CPython hood looked distinctly improvable. egs.
>
> 1. My suggestion to have the docs re. generator-function vs generator-objects
> cleaned up had no takers
> 2. My students trying to work inside the lexer made a mess because the extant lexer is a mess.
> I.e. while python(3) *claims* to accept Unicode input, the actual lexer is
> an ASCII lexer special-cased for unicode rather than pre-lexing utf8 to unicode
>
> These are just specific examples that I am familiar with

Yes, there are some parts of CPython that can be improved. That's true
of every large project (it's said that every program has at least one
bug and could be shortened by at least one instruction, from which it
can be deduced that every program can be reduced to a single
instruction that doesn't work).

Regarding lexers specifically, I have never seen any full-size
language parser that I've wanted to tinker with. They're always highly
optimized pieces of code, dealing with innumerable edge and corner
cases, and exploring them is always like dipping my toe into something
that's either ice-cold water or highly caustic acid, and I can't tell
which.

> Chris' general point still stands, viz take the large and complex program that is cpython
> and clean up these messinesses: You will still have a large and complex program

Right. You could definitely spin off *some* of CPython into a separate
project (flip through the standard library - quite a few of those
modules, if proposed for stdlib inclusion today, would be denied
"better on PyPI"), but my point isn't that it can't be improved, but
that there's an irreducible complexity to it that exceeds the "rewrite
in a quarter" mark by a huge margin.

ChrisA

[toc] | [prev] | [next] | [standalone]

#102011 — Re: Refactoring in a large code base

On Friday, January 22, 2016 at 6:05:02 PM UTC+5:30, Chris Angelico wrote:
> On Fri, Jan 22, 2016 at 11:04 PM, Rustom Mody  wrote:
> > 2. My students trying to work inside the lexer made a mess because the extant lexer is a mess.
> > I.e. while python(3) *claims* to accept Unicode input, the actual lexer is
> > an ASCII lexer special-cased for unicode rather than pre-lexing utf8 to unicode
> >
> > These are just specific examples that I am familiar with
> 
> 
> Regarding lexers specifically, I have never seen any full-size
> language parser that I've wanted to tinker with. They're always highly
> optimized pieces of code, dealing with innumerable edge and corner
> cases, and exploring them is always like dipping my toe into something
> that's either ice-cold water or highly caustic acid, and I can't tell
> which.
> 

You just gave a graphic vivid description...
of the same thing Marko is describing: ;-) viz.
A full-size language parser is something that you - an experienced developer -
make a point of avoiding.
So then the question comes down to this: Is this the order of nature?
Or is it man-made disorder?
Jury's out on that one for lexers/parsers specifically.
For arbitrary code in general, the problem that it may be arbitrarily and unboundedly 
complex/complicated is the oldest problem in computer science: the halting problem.

IOW anyone who thinks that *arbitrary* complexity can *always* be tamed either
has a visa to utopia or needs to re-evaluate (or get) a CS degree

[toc] | [prev] | [next] | [standalone]

#102012 — Re: Refactoring in a large code base

Rustom Mody <rustompmody@gmail.com>:

> IOW anyone who thinks that *arbitrary* complexity can *always* be
> tamed either has a visa to utopia or needs to re-evaluate (or get) a
> CS degree

Not all complexity can be tamed, but what you can't tame you shouldn't
release, either.


Marko

[toc] | [prev] | [next] | [standalone]

#102014 — Re: Refactoring in a large code base

On Friday, January 22, 2016 at 7:13:49 PM UTC+5:30, Marko Rauhamaa wrote:
> Rustom Mody :
> 
> > IOW anyone who thinks that *arbitrary* complexity can *always* be
> > tamed either has a visa to utopia or needs to re-evaluate (or get) a
> > CS degree
> 
> Not all complexity can be tamed, but what you can't tame you shouldn't
> release, either.

And how do you propose to legislate that?
If you leave it to the wetware (untamed!) boxes atop our shoulders will not
two developers have wildly different complexity thresholds?

And as soon as you suggest an objective (∴ algorithmic) solution to detecting 
complexity you have landed with the halting problem (or more precisely Rice theorem)

tl;dr The HP is amazingly deceptive and you just got tripped by it

[toc] | [prev] | [next] | [standalone]

#102016 — Re: Refactoring in a large code base

On Sat, Jan 23, 2016 at 12:30 AM, Rustom Mody <rustompmody@gmail.com> wrote:
> You just gave a graphic vivid description...
> of the same thing Marko is describing: ;-) viz.
> A full-size language parser is something that you - an experienced developer -
> make a point of avoiding.

It's worth noting that "experienced developer" covers a huge range of
skills. There are quite a few other areas that I do not tinker with
(crypto, CPU-level optimizations, and such), not because they're
impossible to understand, but because *I* have not the skill to
understand and improve them. This does mean they're complicated
(they're beyond the "one weekend of tinkering" barrier that any
serious geek should be able to invest), but I'm sure there are
language nerds out there who are so familiar with the grammar of
<insert language here> that they'll pick up CPython's grammar and make
a change with confidence that it'll do what they expect.

> So then the question comes down to this: Is this the order of nature?
> Or is it man-made disorder?
> Jury's out on that one for lexers/parsers specifically.

Lexers/parsers are as complicated as the grammars they parse. A lexer
for a simple structured text file can be pretty easy to implement; for
instance, JSON is pretty straight-forward, with only a handful of
cases (insignificant whitespace, three keywords, two recursive
structures that start with specific characters ('{' and '['), strings
(which start with '"'), and numbers (which start with a digit or a
hyphen)), so a parser need only look for those few possibilities and
it knows exactly what else to fetch up. I could probably write a JSON
parser in a fairly short space of time, and wouldn't be scared of
digging into the internals of someone else's. It's when the grammar
adds complexities to deal with the real-world issues of full size
programming languages that it becomes hairier. The CPython grammar is
only ~150 lines of fairly readable directives, but the parser that
implements it is ~3500 lines of C code. Pike merges the two into a
YACC file of nearly 5000 lines of highly optimized code (it has
different grammar paths for things a human would consider the same, in
order to produce distinct code). That's where I'm ubercautious.

> For arbitrary code in general, the problem that it may be arbitrarily and unboundedly
> complex/complicated is the oldest problem in computer science: the halting problem.
>
> IOW anyone who thinks that *arbitrary* complexity can *always* be tamed either
> has a visa to utopia or needs to re-evaluate (or get) a CS degree

Exactly.

ChrisA

[toc] | [prev] | [next] | [standalone]

#102001 — Re: Refactoring in a large code base

On Fri, 22 Jan 2016 12:19:50 +0200, Marko Rauhamaa wrote:

> We need similar code sanity management. Developers are given much too
> much power to mess up the source code. That's why "legacy" is considered
> a four-letter word among developers.

When I started in this business, in the mid-70s, there was the prospect
of my working under a "programmer-analyst" - there was, then, a whole
hierarchy of programmers.  I resisted that bitterly and was "lucky"
enough to be at the forefront of changes - I would be able to avoid that
until the concept was dead.

Now, 40 years later ... it seems like a good idea to me ... but more
dead than it's ever been and getting deader all the time.

Part of the problem is that the whole profession is dead - now the
people doing the programming are the application experts, and just
programmers are considered at the level that we used to consider
"operators" were at   :)

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | comp.lang.python

csiph-web