o .j@sddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZmZmZdd lmZmZmZdd lmZdd lmZdd lmZmZmZddl m!Z!m"Z"m#Z#m$Z$ddl%m&Z&m'Z'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0ddl1m2Z2ddl3m4Z4m5Z5m6Z6ddlm7Z7m8Z8ddl9m:Z:ej;rddlZ>gdZ?e@eAZBeCdZDe7e8dZEGdddeZFGdd d eFZGd(d&d'ZHdS))) annotationsN) TracebackType)urljoin)HTTPHeaderDict)DEFAULT_BLOCKSIZE)AsyncRequestMethods) _TYPE_BODY_TYPE_BODY_POSITION _TYPE_TIMEOUT ProxyConfig) HttpVersionQuicPreemptiveCacheTypeResponsePromise)port_by_scheme)ProtocolResolver)AsyncBaseResolverAsyncManyResolverAsyncResolverDescription)LocationValueError MaxRetryErrorProxySchemeUnknownURLSchemeUnknown) SSL_KEYWORDSPoolKeykey_fn_by_scheme)AsyncTrafficPolice)connection_requires_http_tunnel)NOT_FORWARDABLE_HEADERS)Retry)UnavailableTraffic)Urlparse_extension parse_url)AsyncHTTPConnectionPoolAsyncHTTPSConnectionPool)AsyncHTTPResponse)Literal)AsyncPoolManagerAsyncProxyManagerproxy_from_url_SelfThttphttpscseZdZUdZdZded<dZded<    d]d^fdd Zd_ddZd`d!d"Z dadbd*d+Z dcd,d-Z  . ddded2d3Z dfd5d6Z dgd9d:Z dadhdsz-AsyncPoolManager.__init__..)super__init__ isinstancerfrom_intcopyr; _num_poolsr@rpoolspool_classes_by_schemer_preemptive_quic_cacher _own_resolverrrSYSTEMstrfrom_url _resolversappendprotocolr _resolver) selfr3r5r7r9r;r?can_resolve_localhostresolver_descriptionrd __class__rGrKrN\s~              zAsyncPoolManager.__init__r^r,cs|SNrGr^rGrGrK __aenter__szAsyncPoolManager.__aenter__exc_typetype[BaseException] | Noneexc_valBaseException | Noneexc_tbTracebackType | NoneLiteral[False]cs|IdHdS)NF)clear)r^rgrirkrGrGrK __aexit__szAsyncPoolManager.__aexit__schemerXhostportrequest_contextdict[str, typing.Any] | Noner%cCs|j|}|dur|j}|ddurt|d<dD]}||dq|dkr3tD]}||dq*|j|d<|j sC|j |_|j|d<d|vrQ|j |d<|||fi|S)a Create a new :class:`urllib3._async.connectionpool.AsyncConnectionPool` based on host, port, scheme, and any additional pool keyword arguments. If ``request_context`` is provided, it is provided as keyword arguments to the pool class used. This method is used to actually create the connection pools handed out by :meth:`connection_from_url` and companion methods. It is intended to be overridden for customization. N blocksize)rprqrrr.r7r9maxsize) rTr;rQgetrpoprrUr] is_availablerecyclerR)r^rprqrrrspool_clskeykwrGrGrK _new_pools"       zAsyncPoolManager._new_poolcs>|jIdH|jr|jr|jIdHdSdSdS)z Empty our store of pools and direct them all to close. This will not affect in-flight connections, but they will not be reused after completion. N)rSrnrVr]ryclosererGrGrKrns zAsyncPoolManager.clearr. str | None int | None pool_kwargscsZ|std||}|pd|d<|st|d}||d<||d<||IdHS)a Get a :class:`urllib3._async.connectionpool.AsyncConnectionPool` based on the host, port, and scheme. If ``port`` isn't given, it will be derived from the ``scheme`` using ``urllib3.connectionpool.port_by_scheme``. If ``pool_kwargs`` is provided, it is merged with the instance's ``connection_pool_kw`` variable and used to create the new connection pool, if one is needed. zNo host specified.r.rprrrqN)r_merge_pool_kwargsrrwlowerconnection_from_context)r^rqrrrprrsrGrGrKconnection_from_hosts  z%AsyncPoolManager.connection_from_hostdict[str, typing.Any]c sbd|vr |d|d}|j|}|durt|\}}ddlm}z|||}Wn ty6Yn]w||}|j|}||d<d|vsN|dsYt |d|d<| }d|vre|dnt } t |dkrt j|vry| t jt j|vr| t jt j|vr| t j| |d<|st|||} |jdur|j|d <|j| |d IdHS) a Get a :class:`urllib3._async.connectionpool.AsyncConnectionPool` based on the request context. ``request_context`` must at least contain the ``scheme`` key and its value must be a key in ``key_fn_by_scheme`` instance variable. strictrpNrload_extensionrr disabled_svnr7rs)rxrrrwr"contrib.webextensions._asyncr ImportErrorscheme_to_http_schemer supported_svnsetlenr h11addh2h3rrUconnection_from_pool_key) r^rsrppool_key_constructor target_schemetarget_implementationr extensionrrpool_keyrGrGrKr.sR                   z(AsyncPoolManager.connection_from_contextrrc s|jjr |j|jj|dd4IdH/}t|ds9|d}|d}|d}|j||||d}||IdHn|}WdIdHn 1IdHsKwY|jdusWJ|j|j_|S) a Get a :class:`urllib3._async.connectionpool.AsyncConnectionPool` based on the provided pool key. ``pool_key`` should be a namedtuple that only contains immutable objects. At a minimum it must have the ``scheme``, ``host``, and ``port`` fields. T)r@Nis_idlerprqrrr)rSbusyreleaselocate_or_holdhasattrr~poolparent)r^rrsswapper_or_poolrprqrrrrGrGrKrqs*  ( z)AsyncPoolManager.connection_from_pool_keyurlcs(t|}|j|j|j|j|dIdHS)a Similar to :func:`urllib3.async_connection_from_url`. If ``pool_kwargs`` is not provided and a new pool needs to be constructed, ``self.connection_pool_kw`` is used to initialize the :class:`urllib3.connectionpool.ConnectionPool`. If ``pool_kwargs`` is provided, it is used instead. Note that if a new pool does not need to be created for the request, the provided ``pool_kwargs`` are not used. rrrprN)r#rrqrrrp)r^rrurGrGrKconnection_from_urls   z$AsyncPoolManager.connection_from_urloverridecs@|j}r|ddDfdd|DS|S)a Merge a dictionary of override values for self.connection_pool_kw. This does not modify self.connection_pool_kw and returns a new dict. Any keys in the override dictionary with a value of ``None`` are removed from the merged dictionary. cSsi|] \}}|dur||qSrdrGrIkvrGrGrK sz7AsyncPoolManager._merge_pool_kwargs..cs*i|]\}}|vs|dur||qSrdrGrrrGrKrs )r;rQupdateitems)r^rbase_pool_kwargsrGrrKrs  z#AsyncPoolManager._merge_pool_kwargs parsed_urlr!boolcCs"|jdurdSt|j|j|j S)z Indicates if the proxy requires the complete destination URL in the request. Normally this is only needed when not using an HTTP CONNECT tunnel. NF)r0rr1rp)r^rrGrGrK!_proxy_requires_url_absolute_forms  z2AsyncPoolManager._proxy_requires_url_absolute_formpromiserResponsePromise | NoneAsyncHTTPResponse | Nonecs|durt|tstdt|dz0|jj|ptddd4IdH}|j|dIdH}WdIdHn 1IdHs>wYWn tyNYdSw|dur[|dur[td|duradSd}|rh|}n|j ryt |j d ry|j j ry|j j }|durtd |j |t t|d }t t|d }|r|rt t|d }t t jtdf|d}t t jtdf|d} t t|d} t t|d} t t jtdf|d} t t|d} |d}t t jtt jf|d}t t|d}t t|d}t t|d}|}t|ts*J|jdkrCd}d}t| } tD]}| |q:z |j||||jd}Wnt yg|j!rc|"IdH|YSw|"IdH|#|IdHt$%d|||j&|t'||df|| || | |d||| | dd |IdH}|j|r|nddIdHSt|j()d}t t|d}|*||j|rt t|d }t t jtdf|d}t t jtdf|d} t t|d} t t|d} t t jtdf|d} t t|d} |d}t t jtt jf|d}t t|d}t t|d}z |j||||jd}Wnt y\|j+rX|"IdH|YSw|"IdH|,|IdHt$%d||j&||df|| || | |d||| | dd |IdH}|j|r|nddIdHS|d }|dur|j-dur|jd!ksd"|jkrd#krn|S|d$kr|.|IdH|S)%z Retrieve the first response available in the pools. This method should be called after issuing at least one request with ``multiplexed=True``. If none available, return None. Nz7get_response only support ResponsePromise but received z] instead. This may occur if you expected the remote peer to support multiplexing but did not.FT)r@ not_idle_onlyrzPInvoked get_response with promise=... that no connections across pools recognize from_promisezPInternal: Unable to identify originating ResponsePromise from a LowLevelResponsemethod pm_redirectpm_urlbodyr5preload_contentdecode_contenttimeoutassert_same_host pool_timeout response_kwchunkedbody_posr?/GETresponse_poolRedirecting %s -> %s) rr5r?rrr release_connrrrr multiplexedz Retry-Afterz Retry: %srei,CONNECT)/rOr TypeErrortyperSborrow get_responser ValueError_fprrforgettypingcastrX get_parameterrget_redirect_locationUnionr rr MutableMappingAnyr rstatusrdiscard incrementrrraise_on_redirect drain_connasync_sleep_for_retrylogdebugurlopenrr5rwis_retryraise_on_status async_sleeprstart_extension)r^rrrrrredirectrrr5rrrrrrrrr?redirect_locationshould_be_removed_header new_promisehas_retry_afterrrGrGrKrsx(                          zAsyncPoolManager.get_responseT.rrrrr}r'cdSrdrGr^rrrrr}rGrGrKr zAsyncPoolManager.urlopen Literal[True]rcrrdrGrrGrGrKrr#AsyncHTTPResponse | ResponsePromisecs*t|}|jdurtjdtddd}|d}|durQt}i}|} t| dkrMt j | vr7| t j t j | vrB| t j t j | vrM| t j ||d<|j|j|j|j|dIdH} |jdur|jd vrdd lm} | t|j}||d<||djd |d <d |d <d|vr|j|d<||r| j||fi|IdH} n| j||jfi|IdH} |j|drt| tr| d|| d|| Sd |d<t| t sJ|o| !} | s| St"|| } | j#dkrd}d|d<t$|d|d<t%D] }|d&|q|d| j'}t|t(s$t(j)||d}|j*rN| +| sN|d,}|dD]}||j*vrH|-|dq8||d<z |j.||| | d}Wnt/yq|j0rm| 1IdH| YSw||d<||d <t23d|| | 1IdH|j|| fi|IdHS)a_ Same as :meth:`urllib3.AsyncHTTPConnectionPool.urlopen` with custom cross-host redirect logic and only sends the request-uri portion of the ``url``. The given ``url`` parameter must be absolute, such that an appropriate :class:`urllib3._async.connectionpool.AsyncConnectionPool` can be chosen for it. Na URLs without a scheme (ie 'https://') are deprecated and will raise an error in a future version of urllib3. To avoid this DeprecationWarning ensure all URLs start with 'https://' or 'http://'. Read more in this issue: https://github.com/urllib3/urllib3/issues/2920r)category stacklevelrrrrr-rFrrr5rrrrrrr?)rrr)4r#rpwarningswarnDeprecationWarningrwrrrr rrrrrrqrrrrrr"rurlopen_kwargsr5rr request_urirSrrOr set_parameterr'rrrrrrr?rrPremove_headers_on_redirect is_same_hostrQrxrrrrrinfo)r^rrrr}rrrr supportedconnrrrrr? new_headersheaderrGrGrKrs                           cCs:ddd|jjD}|r|d7}d||jdS)N; cs|]}t|VqdSrdreprrIprGrGrK 5z,AsyncPoolManager.__repr__.. z)joinrS _registryvaluesr^ inner_reprrGrGrK__repr__4szAsyncPoolManager.__repr__)r2NNN) r3r4r5r6r7r8r9r:r;r<r=r>)r^r,r=r,)rgrhrirjrkrlr=rmrd) rprXrqrXrrr4rsrtr=r%)r=r>Nr.N rqrrrrrprrrtr=r%)rsrr=r%)rrrsrr=r%)rrXrrtr=r%)rrtr=r)rr!r=r)rrr=rT rrXrrXrrrrmr}r<r=r' rrXrrXrrrrr}r<r=r rrXrrXrrr}r<r=rr=rX)__name__ __module__ __qualname____doc__r0__annotations__r1rNrfror~rnrrrrrrrroverloadrr __classcell__rGrGrbrKr)7sL  !  k  1  C&  U  r)cseZdZdZ       d:d;fdd Z   dd?d&d'Zej (d@d)d*dAd1d2Z ej (d@dBd5d2Z (d@dCfd7d2 Z dDd8d9Z Z S)Er*a[ Behaves just like :class:`PoolManager`, but sends all requests through the defined proxy, using the CONNECT method for HTTPS URLs. :param proxy_url: The URL of the proxy to be used. :param proxy_headers: A dictionary containing headers that will be sent to the proxy. In case of HTTP they are being sent with each request, while in the HTTPS/CONNECT case they are sent only once. Could be used for proxy authentication. :param proxy_ssl_context: The proxy SSL context is used to establish the TLS connection to the proxy when using HTTPS proxies. :param use_forwarding_for_https: (Defaults to False) If set to True will forward requests to the HTTPS proxy to be made on behalf of the client instead of creating a TLS tunnel via the CONNECT method. **Enabling this flag means that request and response headers and content will be visible from the HTTPS proxy** whereas tunneling keeps request and response headers and content private. IP address, target hostname, SNI, and port are always visible to an HTTPS proxy even when this flag is disabled. :param proxy_assert_hostname: The hostname of the certificate to verify against. :param proxy_assert_fingerprint: The fingerprint of the certificate to verify against. Example: .. code-block:: python import urllib3 proxy = urllib3.AsyncProxyManager("https://localhost:3128/") resp1 = await proxy.request("GET", "https://google.com/") resp2 = await proxy.request("GET", "https://httpbin.org/") print(len(proxy.pools)) # 1 resp3 = await proxy.request("GET", "https://httpbin.org/") resp4 = await proxy.request("GET", "https://twitter.com/") print(len(proxy.pools)) # 3 r2NF proxy_urlrXr3r4r5r6 proxy_headersproxy_ssl_contextssl.SSLContext | Noneuse_forwarding_for_httpsrproxy_assert_hostnameNone | str | Literal[False]proxy_assert_fingerprintrr;r<r=r>c  st|tr|jd|jd|j} n|} t| } | jdvr#t| j| js3t| jd} | j | d} | |_ |p9i|_ ||_ t |||||_|j | d<|j | d<|j| d<tj||fi| dS) Nz://:r-P)rr_proxy_proxy_headers _proxy_config)rOr%rprqrrr#rrrw_replacer0r'r(r r1rMrN) r^r&r3r5r'r(r*r+r-r; str_proxy_urlr0rrrbrGrKrNts,        zAsyncProxyManager.__init__r.rqrrrrprrtr%csX|dvrtj||||dIdHS|jdusJtj|jj|jj|jj|dIdHS)N>ssewssr/)r)rMrr0rqrrrp)r^rqrrrprrbrGrKrs  z&AsyncProxyManager.connection_from_hostrtyping.Mapping[str, str]cCs0ddi}t|j}|r||d<|r|||S)z Sets headers needed by proxies: specifically, the Accept and Host headers. Only sets headers not provided by the user. Acceptz*/*Host)r#netlocr)r^rr5headers_r:rGrGrK_set_proxy_headerss  z$AsyncProxyManager._set_proxy_headersT.rrrrrmr}r'crrdrGrrGrGrKrrzAsyncProxyManager.urlopenrrcrrdrGrrGrGrKrrrcsZt|}t|j|j|js|d|j}||||d<tj ||fd|i|IdHS)z@Same as HTTP(S)ConnectionPool.urlopen, ``url`` must be absolute.r5rN) r#rr0r1rprwr5r<rMr)r^rrrr}rr5rbrGrKrs  cCsBddd|jjD}|r|d7}d|jd||jdS)Nrcsr rdr r rGrGrKrrz-AsyncProxyManager.__repr__..rzrrrd)rrXr5r6r=r7rrrrr) rr r!r"rNrr<rr$rrr%rGrGrbrKr*=s69,  r*rrXr}r<r=cKstdd|i|S)Nr&rG)r*)rr}rGrGrKr+sr+)rrXr}r<r=r*)I __future__rloggingrrtypesr urllib.parser _collectionsr _constantr_request_methodsr_typingr r r r backendr rrconnectionpoolrcontrib.resolverrcontrib.resolver._asyncrrr exceptionsrrrr poolmanagerrrrutil._async.traffic_policer util.proxyr util.requestr util.retryrutil.traffic_policer util.urlr!r"r#r%r&rr' TYPE_CHECKINGssltyping_extensionsr(__all__ getLoggerrrTypeVarr,rTr)r*r+rGrGrGrKsV                  8