Post by account_disabled on Feb 19, 2024 4:12:55 GMT -5
Below is a summary of the contents of Season 2 Episode 2. Although some parts have been omitted or paraphrased, I have tried to summarize them so that the main meaning is not lost. ① What is crawl budget? Mr. Sanders: Shall we start by defining the crawl budget? GaryLet's summarize what we discussed in the webmaster blog article. Mr. Splitt: good. I think it would be a good idea to read that article first to understand crawl budget. When crawling a website, you must strike a balance between retrieving as much information as possible and not overloading the servers of the site you are crawling. ② What is crawl frequency and necessity of crawling? Mr. Sanders: Is this a crawl limitation? Mr. Splitt: Well, it's the crawl frequency . It means how much load can be placed on the site without causing the server to crash.
We also need to use Google's resources efficiently. The web is so telephone number list large that you need to prioritize it instead of constantly crawling every page. For example, news websites are frequently updated, so you should check them frequently. On the other hand, if your website is about the history of kimchi, for example, it won't be updated often, so you won't need to crawl it frequently. This is called the "need for crawling." Crawl needs affect whether a site is crawled frequently or occasionally. ③Crawling frequency and conditions for determining the necessity of crawling Mr. Sanders: How do you determine whether a crawl is necessary? Mr. Splitt: Crawling a site retrieves information about that site. That information is used for things like deduplication, but it's also used to check when it was last updated.
Webmasters can tell Google when a page has been updated using structured markup or on-page dates, and Google will monitor the frequency of updates. If the update frequency is low, it is determined that there is no need to crawl frequently. This has nothing to do with the quality of the content, even if it's great content that ranks high. It simply determines how often you should visit the site. ④ETag, HTTP header, last updated date of Sitemap.xml, etc. Mr. Sanders: Do you use ETags or last-modified response headers to determine how often your site is crawled? Or are you making decisions based on information obtained from the content? Mr. Splitt: There are various hints that webmasters can provide to Google, including the structured markup I mentioned earlier, ETags and HTTP headers, and the last modified date of Sitemap.xml. However, for example, if you always automatically update the last updated date in Sitemap.
We also need to use Google's resources efficiently. The web is so telephone number list large that you need to prioritize it instead of constantly crawling every page. For example, news websites are frequently updated, so you should check them frequently. On the other hand, if your website is about the history of kimchi, for example, it won't be updated often, so you won't need to crawl it frequently. This is called the "need for crawling." Crawl needs affect whether a site is crawled frequently or occasionally. ③Crawling frequency and conditions for determining the necessity of crawling Mr. Sanders: How do you determine whether a crawl is necessary? Mr. Splitt: Crawling a site retrieves information about that site. That information is used for things like deduplication, but it's also used to check when it was last updated.
Webmasters can tell Google when a page has been updated using structured markup or on-page dates, and Google will monitor the frequency of updates. If the update frequency is low, it is determined that there is no need to crawl frequently. This has nothing to do with the quality of the content, even if it's great content that ranks high. It simply determines how often you should visit the site. ④ETag, HTTP header, last updated date of Sitemap.xml, etc. Mr. Sanders: Do you use ETags or last-modified response headers to determine how often your site is crawled? Or are you making decisions based on information obtained from the content? Mr. Splitt: There are various hints that webmasters can provide to Google, including the structured markup I mentioned earlier, ETags and HTTP headers, and the last modified date of Sitemap.xml. However, for example, if you always automatically update the last updated date in Sitemap.