I wasted too many hours trying to get this to work. Almost every stack overflow post was useless, aside from this one. In my case, I was using Python and Firefox. The following are the steps I had to take in order to get selenium working:
sudo apt-get update
sudo apt-get install firefox-esr
In my case, the most recent supported version is 52.9.0. Yes this is as ancient as it looks, but it will work.
You MUST use a compatible geckodriver. See this page for the current support version. In my case, I needed to use 0.17.
sudo apt-get install xvfb
This is a dependency for the next step:
This was the missing piece for me. You need to have this package and add two additional lines of code.
First add it to your project:
pip install pyvirtualdisplay
Then, add the following lines to your code:
from pyvirtualdisplay import Display
display = Display(visible=0, size=(800, 600))
display.start()
from pyvirtualdisplay import Display
# Configure virtual display
display = Display(visible=0, size=(800, 600))
display.start()
# Handle scrape via selenium
options = Options()
options.headless = True
driver = webdriver.Firefox(options=options, service_log_path=devnull)
driver.get(url)
# Do stuff here...
driver.quit()