Every time a page a requested, favicon is also downloaded if present at the server. Though favicon download doesn't affect you page response time directly, but it share the bandwidth. So better to keep favicon small (preferably less than 1 KB) & cacheable.
Favicon can be used in two ways:
Keeping the favicon.ico file in the server root directory
favicon.ico using link attribute in the header of the HTML
I consider second approach better for the following reasons
can be cached with the proper expiry header and can be served from CDN.
<link rel="shortcut icon" href="http://hiox.org/favicon.ico" type="image/x-icon">
Please note that, If favicon is changed, It name cann't be changed to clear the cache, so choose the expiry time wisely.
However there are something more to know about the second approach:
IE loads favicon before lazy-loaded components.
It increases the load on your application server.
Favicon loaded with a <link> are loaded early in the FF waterfall