Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #63935 > unrolled thread

fork seems to make urlopen into a black hole?

Back to article view | Back to comp.lang.python

Contents

  fork seems to make urlopen into a black hole? BobAalsma <overhaalsgang_24_bob@me.com> - 2014-01-14 12:04 -0800
    Re: fork seems to make urlopen into a black hole? Chris Angelico <rosuav@gmail.com> - 2014-01-15 07:07 +1100

#63935 — fork seems to make urlopen into a black hole?

A program took much too long to check some texts collected from web pages.
As this could be made parallel easily, I put in fork.
And the result seems to be that the program simply stops in the line with urlopen. Any suggestions?

Relevant part:

try:
            print 'urlopen by', kind_nummer, '- before'
            extern_adres = urlopen(adres).geturl()
            print 'urlopen by', kind_nummer, '- after'
        except IOError:
            tekst_string = 'URL "' + adres +'" geeft IO Error'
            print tekst_string, 'door', kind_nummer
            bloklijst_verslag.append(tekst_string + '\n')
            tekst_string = '\t**** Deze pagina wordt overgeslagen\n'
            bloklijst_verslag.append(tekst_string + '\n')
            adres = ''
        except:
            fout = sys.exc_info()[:2]
            tekst_string = 'URL "' + adres + '" geeft fout ' + fout
            print tekst_string, 'door', kind_nummer
            bloklijst_verslag.append(tekst_string + '\n')
            tekst_string = '\t**** Deze pagina wordt overgeslagen\n'
            bloklijst_verslag.append(tekst_string + '\n')
            adres = ''
        else:
            print 'urlopen by', kind_nummer, '- else'
            if extern_adres != adres:
                tekst_string = '\t**** redirect naar ' + extern_adres
                bloklijst_verslag.append(tekst_string + '\n')

From this block, the only response seen is the first print statement ("url open by .... - before") and then nothing else seems to happen in that child.
This happens for all children: the expected number of children is reached and they all reach (only) this point.

[toc] | [next] | [standalone]

#63936

On Wed, Jan 15, 2014 at 7:04 AM, BobAalsma <overhaalsgang_24_bob@me.com> wrote:
> A program took much too long to check some texts collected from web pages.
> As this could be made parallel easily, I put in fork.

Rather than using the low-level fork() function, you may find it
easier to manage things if you use the multiprocessing module. I don't
know if it'll help in this specific case, but if nothing else, it'll
be easier to see what's going on.

ChrisA

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web