Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #100076

Problems using celery and pyelasticsearch

Show all headers | View raw

Hi,

I create a task that uses this function

def get_jobs(specialization, yoe):
     try:
         key = "SpecAlert-"
         if len(specialization):
             key += '-'.join(map(str, specialization))

         if yoe is not None:
             key += '-yoe'.join(yoe)

         jobs = memcache_client.get(key)

         if not jobs:
             spec_param = {'terms': {'jobs.specialization.id': 
specialization}}
             if yoe is not None:
                 if len(yoe) == 2:
                     yoe_param = {'range': {'jobs.experience': {'gte': 
yoe[0], 'lte': yoe[1]}}}
                 elif int(yoe[0]):
                     yoe_param = {'range': {'jobs.experience': {'gte': 
yoe[0]}}}
                 else:
                     yoe_param = {'term': {'jobs.experience': yoe[0]}}
             query_bool = {'must': [{'range': {'jobs.deadline': {'gt': 
str(date.today() + timedelta(days=1))}}}]}
             query_bool['must_not'] = [{'term': {'jobs.job_level': 
JOB_LEVEL_VOC}}]
             if specialization:
                 query_bool['must'].append(spec_param)
             if yoe:
                 query_bool['must'].append(yoe_param)
             es = config.get_es_connection()
             es_config = config.config['elasticsearch']
             # print({'query': {'bool': query_bool}})
             try:
                 # Tasks sometimes hang here
                 result = es.search(index=es_config['job_index'], 
doc_type=es_config['job_type'],
                                     body={'query': {'bool': query_bool}})
                 jobs = []
                 for j in result['hits']['hits']:
                     jobs.append(j['_source'])
             except ElasticsearchException as esc:
                 print(esc)
                 jobs = []

             if jobs:
                 memcache_client.set(key, jobs, 3600)
     except Exception as e:
         jobs = []
         print(e)

     return jobs

I find that the celery worker often stops executing tasks. After tests 
and debugging I in that this NEVER happens when I take out this line(s):
result = es.search(index=es_config['job_index'], 
doc_type=es_config['job_type'],
                                     body={'query': {'bool': query_bool}})

This line also does not raise any Exceptions

Does anyone have any idea what could be going on or how I can further 
inspect running tasks.
N.B celery worker is started with loglevel=debug flag but does not 
output any useful info as regards the problem.

Thanks

Back to comp.lang.python | Previous | Next | Find similar | Unroll thread

Thread

Problems using celery and pyelasticsearch nonami <nonami.ayo@gmail.com> - 2015-12-07 02:37 +0100

csiph-web