web scraping by selenium pyhon3

For new starter as me.

Assume everyone knows how to install selenium and how to write code with python 3 and the principle of HTML. Selenium is an automated browser package, and also a chemical element(introduced below)

Selenium is a chemical element with symbol Se and atomic number 34. It is a nonmetal with properties that are intermediate between the elements above and below in the periodic table, sulfur and tellurium. It rarely occurs in its elemental state or as pure ore compounds in the Earth's crust. Selenium (Greek σελήνη selene meaning "Moon") was discovered in 1817 by Jöns Jacob Berzelius, who noted the similarity of the new element to the previously discovered tellurium (named for the Earth).(refer from wiki)

And there is a website to introduce how to write selenium with python:

http://selenium-python.readthedocs.io/

1.Before start programming, those pre-setting procedures shall be down.

1.1.install selenium:

it can be done by pip or easy_install

1.2.download web driver

it depends on what browsers you use, in my case, I use chrome, so google chrome driver is what I need.

1.2.1 download web driver

you can select the corresponding type(for windows,mac or linux...etc) driver from this site:

https://sites.google.com/a/chromium.org/chromedriver/downloads

1.2.2 unzip the downloaded file to the selected folder, which you can freely set. 

2.The web scraping procedures are descript below:

2.0 import lib

..webdriver is web driver

..support.ui import Select is for drop down selection or other  item selectons

..Keys is for mouse/ keyboard action, like click ,right click ,string input...etc.

2.1 Set webdriver

it is needed to inform selenium where is the web driver. Path is needed.

Then input the right type browser command.

2.2.Get URL

the driver will connect to the website

2.3 Change text 

To key in the date I want:

First find the element of date item, so I check the source of the website:

The name of element is "datestart" , so let driver find element by its name,

And then clear the value of the element and refill with the value I want.

Then click the element.

Before

After

 2.4 Select the item of Contract:

Select the selection attribute item from website

and print all the options in selection

Then select the first item of all options. 

   

the items in selection

 Before

After, the Contract item is changed

how to call pip on windows

For new starter as me.
pip is PyPA recommended tool for installing Python packages. so if you want install some packages, you can use this.
1. initialize command line:

2. key in "cd" command and key in the needed directory/file location. In my case , I need to initialize pip, so I redirect to pip folder.

3. key in or paste the folder location

4. press Enter, then the system will redirect to the folder

5. you can call any file in this folder, in my case, I just call "pip"

6. press Enter, then the pip will run. the result as figure shown below.

7. Because I need to install beautifulsuop4 to do some web crawling application, so I key in "pip install beautifulsoup4"

8. press Enter, then the install process will run as figure shown below:

9. According to the figure , the process is successful.

Appendix:

Beautiful soup website:

https://www.crummy.com/software/BeautifulSoup/bs4/doc/