Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #73374 > unrolled thread

pypi - package metdata

Back to article view | Back to comp.lang.python

Contents

  pypi - package metdata Maciej Dziardziel <fiedzia@gmail.com> - 2014-06-18 12:21 -0700
    Re: pypi - package metdata Maciej Dziardziel <fiedzia@gmail.com> - 2014-06-18 12:24 -0700
      Re: pypi - package metdata Mark Lawrence <breamoreboy@yahoo.co.uk> - 2014-06-18 21:12 +0100
        Re: pypi - package metdata Maciej Dziardziel <fiedzia@gmail.com> - 2014-06-18 14:05 -0700
      Re: pypi - package metdata Terry Reedy <tjreedy@udel.edu> - 2014-06-18 16:39 -0400

#73374 — pypi - package metdata

I wasn't quiet happy with the way search on pypi works,
so I've got an idea of getting all package metadata from pypi
and do search locally. The only problem is that I can't figure out
where to get the data from. I tried to use bandersnatch to 
set up mirror, but all I've got was a mixture of egg, whl and exe files.
Egg and whl are useful maybe, but I guess its not exactly what I want.
Then I've found api:
https://pypi.python.org/simple/ provides list of packages,
and I can use json api to get metadata for each of them.
 

[toc] | [next] | [standalone]

#73375

On Wednesday, June 18, 2014 8:21:12 PM UTC+1, Maciej Dziardziel wrote:
> I wasn't quiet happy with the way search on pypi works,
> 
> so I've got an idea of getting all package metadata from pypi
> 
> and do search locally. The only problem is that I can't figure out
> 
> where to get the data from. I tried to use bandersnatch to 
> 
> set up mirror, but all I've got was a mixture of egg, whl and exe files.
> 
> Egg and whl are useful maybe, but I guess its not exactly what I want.
> 
> Then I've found api:
> 
> https://pypi.python.org/simple/ provides list of packages,
> 
> and I can use json api to get metadata for each of them.


(Click post to fast, so I am continuing.)

So its doable, but I am really hoping for a better way.
I assume pypi would ban me quickly for making thousands of requests,
and I'd love to be able to update index daily or weekly.

Is there something I am missing?

[toc] | [prev] | [next] | [standalone]

#73378

On 18/06/2014 20:24, Maciej Dziardziel wrote:
> On Wednesday, June 18, 2014 8:21:12 PM UTC+1, Maciej Dziardziel wrote:
>> I wasn't quiet happy with the way search on pypi works,
>>
>> so I've got an idea of getting all package metadata from pypi
>>
>> and do search locally. The only problem is that I can't figure out
>>
>> where to get the data from. I tried to use bandersnatch to
>>
>> set up mirror, but all I've got was a mixture of egg, whl and exe files.
>>
>> Egg and whl are useful maybe, but I guess its not exactly what I want.
>>
>> Then I've found api:
>>
>> https://pypi.python.org/simple/ provides list of packages,
>>
>> and I can use json api to get metadata for each of them.
>
>
> (Click post to fast, so I am continuing.)
>
> So its doable, but I am really hoping for a better way.
> I assume pypi would ban me quickly for making thousands of requests,
> and I'd love to be able to update index daily or weekly.
>
> Is there something I am missing?
>

Why not use google and do a site specific search of pypi?

Slight aside would you please use the mailing list 
https://mail.python.org/mailman/listinfo/python-list or read and action 
this https://wiki.python.org/moin/GoogleGroupsPython to prevent us 
seeing double line spacing and single line paragraphs, thanks.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask 
what you can do for our language.

Mark Lawrence

---
This email is free from viruses and malware because avast! Antivirus protection is active.
http://www.avast.com

[toc] | [prev] | [next] | [standalone]

#73381

On Wednesday, June 18, 2014 9:12:46 PM UTC+1, Mark Lawrence wrote:
> Why not use google and do a site specific search of pypi?

I am looking for good use of elasticsearch I have there,
rather then quick way of finding something once quickly.
Neither pypi nor google provide faceting for example.

I've found that (confusingly) json api for pypi doesn't do anything useful,
but xmlrpc provides more options, and its good enough for me.

[toc] | [prev] | [next] | [standalone]

#73380

On 6/18/2014 3:24 PM, Maciej Dziardziel wrote:
> On Wednesday, June 18, 2014 8:21:12 PM UTC+1, Maciej Dziardziel wrote:
>> I wasn't quiet happy with the way search on pypi works,
>>
>> so I've got an idea of getting all package metadata from pypi
>>
>> and do search locally. The only problem is that I can't figure out
>>
>> where to get the data from. I tried to use bandersnatch to
>>
>> set up mirror, but all I've got was a mixture of egg, whl and exe files.
>>
>> Egg and whl are useful maybe, but I guess its not exactly what I want.
>>
>> Then I've found api:
>>
>> https://pypi.python.org/simple/ provides list of packages,
>>
>> and I can use json api to get metadata for each of them.
>
>
> (Click post to fast, so I am continuing.)
>
> So its doable, but I am really hoping for a better way.
> I assume pypi would ban me quickly for making thousands of requests,
> and I'd love to be able to update index daily or weekly.
>
> Is there something I am missing?

I believe there is an API for updates, that is meant for mirrors. But I 
agree with Mark that you should try google first



-- 
Terry Jan Reedy

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web