How to get all the links from a web page

In this article, I will show how to get all the links from a webpage, maybe not just link, but I will use the links as an example.

We just take Google as an example and my browser is Chrome, for example I just search web 3.0 from Google, and from my browser, the result is like below:

In order to get the links from this page, first I will investigate the links from source code, we can open the Inspect Elements window with one of the following method:

1. Open from menu View/Developer/Inspect Elements
2. Open with shortcut ⌥+⌘+C
3. Right click the page and select Inspect context menu

Then select one of the link to see the source code, like below:

We can see that all the links are located under <div class="yuRUbf"> , so I can get all the a element under div with classyuRubf .

Then select the Console tab from Inspect Elementswindow like below:

In the console, we can get the links with the following code:var links = document.querySelectorAll('.yuRUbf>a');
var urls = [];
for (let link of links) {
   urls.push(link.getAttribute('href'));
}
console.log(urls.join('\n'));

The output is like below:

In order to get other information, you can change the code accordingly, for example, I also want to get the title of the link, from the source code, we can see that the title is under h3 tag, which is inside the a tag, then we can use the following code to get the titles.var links = document.querySelectorAll('.yuRUbf>a');
var urls = [];
for (let link of links) {
   urls.push(link.getElementsByTagName('h3')[0].innerText + '\t' + link.getAttribute('href'));
}
console.log(urls.join('\n'));

The result is like below:

Hope this article can help you when you want to get some information from a web page.