Csdrhrt

I'm trying to webscrap a web page with CloudFlare protection from remote server:

#!/usr/bin/env python3



from xvfbwrapper import Xvfb

from selenium import webdriver

from selenium.webdriver.chrome.options import Options

import time

import logging

display = Xvfb()

display.start()



try:

    logging.basicConfig(format='%(asctime)s - %(name)s - %(levelname)s - %(message)s', level=logging.INFO)

    logger = logging.getLogger('main')

    chrome_options = Options()

    chrome_options.add_argument("--no-sandbox")

    chrome_options.add_argument("--disable-setuid-sandbox")

    browser = webdriver.Chrome(chrome_options=chrome_options)

    browser.get('url ...')

    time.sleep(10)

    logger.info('woke up')

    logger.info(browser.get_cookies())

finally:

    display.stop()

The problem is that call of get_cookies blocking for exactly 4 minutes. Also if I'm trying to continue traverse web site I'm also stucking on every request to server for exactly 4 minutes:

root@server:~# ./s.py

2019-02-05 00:52:11,509 - main - INFO - woke up

2019-02-05 00:56:06,506 - main - INFO - [some cookies ...]

I tried to kill script while it's stuck and see that code stucks here:

root@server:~# ./s.py

2019-02-05 01:00:31,333 - main - INFO - woke up

^CTraceback (most recent call last):

  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request

    httplib_response = conn.getresponse(buffering=True)

TypeError: getresponse() got an unexpected keyword argument 'buffering'



During handling of the above exception, another exception occurred:



Traceback (most recent call last):

  File "./s.py", line 22, in <module>

logger.info(browser.get_cookies())

  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 841, in get_cookies

    return self.execute(Command.GET_ALL_COOKIES)['value']

  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/webdriver.py", line 319, in execute

    response = self.command_executor.execute(driver_command, params)

  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py", line 374, in execute

    return self._request(command_info[0], url, body=data)

  File "/usr/local/lib/python3.6/dist-packages/selenium/webdriver/remote/remote_connection.py", line 397, in _request

    resp = self._conn.request(method, url, body=body, headers=headers)

  File "/usr/lib/python3/dist-packages/urllib3/request.py", line 66, in request

**urlopen_kw)

  File "/usr/lib/python3/dist-packages/urllib3/request.py", line 87, in request_encode_url

return self.urlopen(method, url, **extra_kw)

  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 321, in urlopen

response = conn.urlopen(method, u.request_uri, **kw)

  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen

chunked=chunked)

  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request

    httplib_response = conn.getresponse()

  File "/usr/lib/python3.6/http/client.py", line 1331, in getresponse

response.begin()

  File "/usr/lib/python3.6/http/client.py", line 297, in begin

version, status, reason = self._read_status()

  File "/usr/lib/python3.6/http/client.py", line 258, in _read_status

    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")

  File "/usr/lib/python3.6/socket.py", line 586, in readinto

return self._sock.recv_into(b)

KeyboardInterrupt

Also I tried to attach to script with strace and found out that script just blocked on getting packet from socket:

root@server:~# strace -p 7545

strace: Process 7545 attached

recvfrom(4, 0x1ec58f0, 8192, 0, NULL, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)

I tried that with different versions of Chrome/Selenium and got always the same result: sticking for 4 minutes.

Ubuntu 18.04.1 LTS

Google Chrome 66.0.3359.181

ChromeDriver 2.38.552522 (437e6fbedfa8762dec75e2c5b3ddb86763dc9dcb)

Name: selenium Version: 3.141.0

Name: xvfbwrapper Version: 0.2.9