£Á°èZ¨Ä…–K§‚«“ô4“ÒÙ´dîfUÙÃÅ WKbyʦ•êŽ…È®FÒ¿ÊÎóCozá¬S@6{Í:›œêZÌ:Š•_%:¢¾¾~;‘Ã~芩ÊǍí`ÔÑ©ú뙵'5I¿fš×WO%ø9¾«¾DK|€ùÍD”Ýs]nHÕ¶êםӼ㞪éUWŸÈË%DÒÕ¬ï‘]/Åcx ‰ï2ß]ä6G[]S£Ôϯrs{úëóµmÒï#UQxo·õÞCe]"±/aÙ&Eã4ú9Jé_ÞåëdãöKë)AÞ ¯¹ægƒÛowЍø^d™ý½ßB7áyMä9ÜÖUã !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! B  fh@s0ddgZddlZddlZddlZddlZddlZddlZddlZddlm Z ddlm Z m Z dZ dZ dZeZdd Zd d ZGd d d eZGdddZddZGdddeZd$ddZddZGdddeZGdddeZeZGdddeZGdd d eZGd!d"d"eZ Gd#ddeZ!dS)%Pool ThreadPoolN)util) get_context TimeoutErrorcCs tt|S)N)listmap)argsr 9/opt/alt/python37/lib64/python3.7/multiprocessing/pool.pymapstar+srcCstt|d|dS)Nrr)r itertoolsstarmap)r r r r starmapstar.src@seZdZddZddZdS)RemoteTracebackcCs ||_dS)N)tb)selfrr r r __init__6szRemoteTraceback.__init__cCs|jS)N)r)rr r r __str__8szRemoteTraceback.__str__N)__name__ __module__ __qualname__rrr r r r r5src@seZdZddZddZdS)ExceptionWithTracebackcCs0tt|||}d|}||_d||_dS)Nz """ %s""") tracebackformat_exceptiontypejoinexcr)rr rr r r r<s zExceptionWithTraceback.__init__cCst|j|jffS)N) rebuild_excr r)rr r r __reduce__Asz!ExceptionWithTraceback.__reduce__N)rrrrr"r r r r r;srcCst||_|S)N)r __cause__)r rr r r r!Ds r!cs0eZdZdZfddZddZddZZS)MaybeEncodingErrorzVWraps possible unpickleable errors, so they can be safely sent through the socket.cs.t||_t||_tt||j|jdS)N)reprr valuesuperr$r)rr r&) __class__r r rPs  zMaybeEncodingError.__init__cCsd|j|jfS)Nz(Error sending result: '%s'. Reason: '%s')r&r )rr r r rUszMaybeEncodingError.__str__cCsd|jj|fS)Nz<%s: %s>)r(r)rr r r __repr__YszMaybeEncodingError.__repr__)rrr__doc__rrr) __classcell__r r )r(r r$Ls r$r Fc Cs|dk r(t|tr|dks(td||j}|j}t|drR|j|j |dk rb||d}xP|dks|r||kry |} Wn$t t fk rt dPYnX| dkrt dP| \} } } } }yd| | |f}WnHtk r.}z(|r| tk rt||j}d|f}Wdd}~XYnXy|| | |fWnRtk r}z2t||d}t d ||| | d|ffWdd}~XYnXd} } }} } }|d7}qjWt d |dS) NrzMaxtasks {!r} is not valid_writerrz)worker got EOFError or OSError -- exitingzworker got sentinel -- exitingTFz0Possible encoding error while sending result: %szworker exiting after %d tasks) isinstanceintAssertionErrorformatputgethasattrr,close_readerEOFErrorOSErrorrdebug Exception_helper_reraises_exceptionr __traceback__r$)inqueueoutqueue initializerinitargsZmaxtasksZwrap_exceptionr1r2Z completedtaskjobifuncr kwdsresultewrappedr r r worker]sJ       $ rHcCs|dS)z@Pickle-able helper function for use by _guarded_task_generation.Nr )Zexr r r r:sr:c@s,eZdZdZdZddZd>ddZd d Zd d Zd dZ ddZ difddZ d?ddZ d@ddZ dAddZddZdBddZdCddZdiddfd d!ZdDd"d#ZdEd$d%Zed&d'Zed(d)Zed*d+Zed,d-Zd.d/Zd0d1Zd2d3Zd4d5Zed6d7Zed8d9Z d:d;Z!dr?Zmaxtasksperchildcontextr r r rsT         z Pool.__init__cCsZd}xPttt|jD]:}|j|}|jdk rtd||d}|j|=qW|S)zCleanup after any worker processes which have exited due to reaching their specified lifetime. Returns True if any workers were cleaned up. FNzcleaning up worker %dT)reversedrangelenr]exitcoderr8r)rZcleanedrBrHr r r _join_exited_workerss   zPool._join_exited_workersc Cs|xvt|jt|jD]^}|jt|j|j|j|j |j |j fd}|j ||j dd|_ d|_|tdqWdS)zBring the number of pool processes up to the specified number, for use after reaping workers which have exited. )rKr rJZ PoolWorkerTz added workerN)rqr\rrr]rJrHrlrfrUrVrT_wrap_exceptionappendnamereplacerbrcrr8)rrBwr r r r^s  zPool._repopulate_poolcCs|r|dS)zEClean up any exited workers and start replacements for them. N)rtr^)rr r r _maintain_poolszPool._maintain_poolcCs4|j|_|j|_|jjj|_|jjj|_ dS)N) rIrOrlrfr,sendrer5recvri)rr r r rMs   zPool._setup_queuescCs||||S)zT Equivalent of `func(*args, **kwds)`. Pool must be running. ) apply_asyncr2)rrCr rDr r r applysz Pool.applycCs|||t|S)zx Apply `func` to each element in `iterable`, collecting the results in a list that is returned. ) _map_asyncrr2)rrCiterable chunksizer r r r szPool.mapcCs|||t|S)z Like `map()` method but the elements of the `iterable` are expected to be iterables as well and will be unpacked as arguments. Hence `func` and (a, b) becomes func(a, b). )rrr2)rrCrrr r r rsz Pool.starmapcCs|||t|||S)z= Asynchronous version of `starmap()` method. )rr)rrCrrcallbackerror_callbackr r r starmap_asyncs zPool.starmap_asyncc csny0d}x&t|D]\}}||||fifVqWWn8tk rh}z||dt|fifVWdd}~XYnXdS)zProvides a generator of tasks for imap and imap_unordered with appropriate handling for iterables which throw exceptions during iteration.rN) enumerater9r:)rZ result_jobrCrrBxrFr r r _guarded_task_generations zPool._guarded_task_generationrcCs|jtkrtd|dkrFt|j}|j||j|||j f|S|dkr\td |t |||}t|j}|j||jt ||j fdd|DSdS)zP Equivalent of `map()` -- can be MUCH slower than `Pool.map()`. zPool not runningrzChunksize must be 1+, not {0:n}css|]}|D] }|Vq qdS)Nr ).0chunkitemr r r EszPool.imap..N)rSrRrY IMapIteratorrQrPr1r_job _set_lengthr0r _get_tasksr)rrCrrrE task_batchesr r r imap)s(     z Pool.imapcCs|jtkrtd|dkrFt|j}|j||j|||j f|S|dkr\td |t |||}t|j}|j||jt ||j fdd|DSdS)zL Like `imap()` method but ordering of results is arbitrary. zPool not runningrzChunksize must be 1+, not {0!r}css|]}|D] }|Vq qdS)Nr )rrrr r r rbsz&Pool.imap_unordered..N)rSrRrYIMapUnorderedIteratorrQrPr1rrrr0rrr)rrCrrrErr r r imap_unorderedGs&      zPool.imap_unorderedcCsB|jtkrtdt|j||}|j|jd|||fgdf|S)z; Asynchronous version of `apply()` method. zPool not runningrN)rSrRrY ApplyResultrQrPr1r)rrCr rDrrrEr r r r}ds  zPool.apply_asynccCs|||t|||S)z9 Asynchronous version of `map()` method. )rr)rrCrrrrr r r map_asyncoszPool.map_asyncc Cs|jtkrtdt|ds$t|}|dkrTtt|t|jd\}}|rT|d7}t|dkrdd}t |||}t |j |t|||d} |j || j||df| S)zY Helper function to implement map, starmap and their async counterparts. zPool not running__len__Nrr)r)rSrRrYr3r divmodrrr]rr MapResultrQrPr1rr) rrCrZmapperrrrZextrarrEr r r rws&   zPool._map_asynccCsTt}x0|jtks$|jr8|jtkr8|tdq W|j dt ddS)Ng?zworker handler exiting) r_current_threadrSrRrQ TERMINATErztimesleeprPr1rr8)poolthreadr r r r`s  zPool._handle_workersc Csjt}xt|jdD]\}}d}zx|D]}|jr@tdPy ||Wq*tk r} zB|dd\} } y|| | d| fWnt k rYnXWdd} ~ XYq*Xq*W|rtd|r|dnd} || dwPWdd}}} XqWtdy:td| dtd x|D]} |dq&WWn t k rZtd YnXtd dS) Nz'task handler found thread._state != RUNrFzdoing set_length()rrztask handler got sentinelz/task handler sending sentinel to result handlerz(task handler sending sentinel to workersz/task handler got OSError when sending sentinelsztask handler exiting) r_riterr2rSrr8r9_setKeyErrorr1r7) taskqueuer1r=rcacherZtaskseqZ set_lengthr@rFrAidxpr r r rdsB          zPool._handle_tasksc Cst}xy |}Wn"ttfk r6tddSX|jr\|jtksPtdtdP|dkrptdP|\}}}y|| ||Wnt k rYnXd}}}q Wx|rP|jtkrPy |}Wn"ttfk rtddSX|dkr tdq|\}}}y|| ||Wnt k r@YnXd}}}qWt |drtdy,x&t dD]}|j sP|qtWWnttfk rYnXtd t||jdS) Nz.result handler got EOFError/OSError -- exitingzThread not in TERMINATEz,result handler found thread._state=TERMINATEzresult handler got sentinelz&result handler ignoring extra sentinelr5z"ensuring that outqueue is not full z7result handler exiting: len(cache)=%s, thread._state=%s)r_rr7r6rr8rSrr/rrr3rqr5pollrr)r=r2rrr@rArBobjr r r rhs\             zPool._handle_resultsccs4t|}x&tt||}|s"dS||fVq WdS)N)rtuplerislice)rCitsizerr r r r s zPool._get_taskscCs tddS)Nz:pool objects cannot be passed between processes or pickled)NotImplementedError)rr r r r"szPool.__reduce__cCs&td|jtkr"t|_t|j_dS)Nz closing pool)rr8rSrRCLOSEra)rr r r r4s  z Pool.closecCs$tdt|_t|j_|dS)Nzterminating pool)rr8rrSrarm)rr r r terminate s zPool.terminatecCsntd|jtkrtdn|jttfkr4td|j|j |j x|j D] }|qZWdS)Nz joining poolzPool is still runningzIn unknown state) rr8rSrRrYrrrarrgrjr])rrr r r r&s       z Pool.joincCsDtd|jx*|r>|jr>|jt dqWdS)Nz7removing tasks from inqueue until task handler finishedr) rr8Z_rlockacquireis_aliver5rr|rr)r< task_handlerrr r r _help_stuff_finish2s    zPool._help_stuff_finishc CsJtdt|_t|_td|||t||sNt|dkrNtdt|_|dtdt |k r|| |rt |ddrtdx|D]} | j dkr| qWtdt |k r| td t |k r| |rFt |ddrFtd x0|D](} | rtd | j| qWdS) Nzfinalizing poolz&helping task handler/workers to finishrz.Cannot have cache with result_hander not alivezjoining worker handlerrzterminating workerszjoining task handlerzjoining result handlerzjoining pool workerszcleaning up worker %d)rr8rrSrrrrr/r1r_rrr3rsrpid) clsrr<r=rZworker_handlerrZresult_handlerrrr r r rk;s<                zPool._terminate_poolcCs|S)Nr )rr r r __enter__kszPool.__enter__cCs |dS)N)r)rexc_typeZexc_valZexc_tbr r r __exit__nsz Pool.__exit__)NNr NN)N)N)NNN)r)r)NNN)NNN)#rrrr*rurJrrtr^rzrMr~r rrrrrr}rr staticmethodr`rdrhrr"r4rrr classmethodrkrrr r r r rsF 8         . <  0c@s@eZdZddZddZddZddd Zdd d Zd d ZdS)rcCs4t|_tt|_||_||_||_|||j<dS)N) r_ZEvent_eventnext job_counterrrQ _callback_error_callback)rrrrr r r rws   zApplyResult.__init__cCs |jS)N)rZis_set)rr r r readyszApplyResult.readycCs|std||jS)Nz{0!r} not ready)rrYr0_success)rr r r successfulszApplyResult.successfulNcCs|j|dS)N)rwait)rtimeoutr r r rszApplyResult.waitcCs,|||st|jr"|jS|jdS)N)rrrr_value)rrr r r r2s  zApplyResult.getcCsT|\|_|_|jr$|jr$||j|jr<|js<||j|j|j|j=dS)N)rrrrrsetrQr)rrBrr r r rs      zApplyResult._set)N)N) rrrrrrrr2rr r r r rus   rc@seZdZddZddZdS)rcCsftj||||dd|_dg||_||_|dkrLd|_|j||j=n||t |||_dS)N)rTr) rrrr _chunksize _number_leftrrrbool)rrrlengthrrr r r rs    zMapResult.__init__cCs|jd8_|\}}|rp|jrp||j||j|d|j<|jdkr|jrZ||j|j|j=|jnF|s|jrd|_||_|jdkr|j r| |j|j|j=|jdS)NrrF) rrrrrrQrrrr)rrBZsuccess_resultsuccessrEr r r rs"         zMapResult._setN)rrrrrr r r r rs rc@s:eZdZddZddZd ddZeZdd Zd d ZdS) rcCsJtt|_tt|_||_t |_ d|_ d|_ i|_ |||j<dS)Nr)r_Z ConditionZLock_condrrrrQ collectionsdeque_items_index_length _unsorted)rrr r r rs  zIMapIterator.__init__cCs|S)Nr )rr r r __iter__szIMapIterator.__iter__Nc Cs|jy|j}Wnntk r|j|jkr8td|j|y|j}Wn,tk r~|j|jkrttdtdYnXYnXWdQRX|\}}|r|S|dS)N) rrpopleft IndexErrorrr StopIterationrr)rrrrr&r r r rs"   zIMapIterator.nextc Cs|j|j|krr|j||jd7_x8|j|jkrd|j|j}|j||jd7_q.W|jn ||j|<|j|jkr|j|j =WdQRXdS)Nr) rrrrvrpopnotifyrrQr)rrBrr r r rs      zIMapIterator._setc Cs<|j,||_|j|jkr.|j|j|j=WdQRXdS)N)rrrrrQr)rrr r r rs   zIMapIterator._set_length)N) rrrrrr__next__rrr r r r rs   rc@seZdZddZdS)rc CsP|j@|j||jd7_|j|j|jkrB|j|j=WdQRXdS)Nr)rrrvrrrrQr)rrBrr r r r s    zIMapUnorderedIterator._setN)rrrrr r r r r src@s:eZdZdZeddZd ddZdd Zed d ZdS) rFcOsddlm}|||S)Nr)rJ)ZdummyrJ)r rDrJr r r rJs zThreadPool.ProcessNr cCst||||dS)N)rr)rrnr>r?r r r r!szThreadPool.__init__cCs,t|_t|_|jj|_|jj|_dS)N)rNrOrlrfr1rer2ri)rr r r rM$s   zThreadPool._setup_queuescCsNyx|jddqWWntjk r,YnXxt|D]}|dq8WdS)NF)block)r2rNZEmptyrqr1)r<rrrBr r r r*szThreadPool._help_stuff_finish)NNr ) rrrrurrJrrMrr r r r rs   )Nr NF)"__all__r_rNrrrWrrrrrrrRrrcountrrrr9rrr!r$rHr:objectrrZ AsyncResultrrrrr r r r  s@   ,f')@