spiderja.htm: How to search the web, by fravia+ spiderja

~ Essays ~

				essays

(Courtesy of fravia's advanced searching lores)

A Tale of the Spider in a Jar
by Servo
published at fravia's searchlores in february 2000

Here is Servo's email to me:

      
> > Fravia+, 
> > 
> > Enclosed is a writing of spiders restricted to small topics of the web 
> > or intranets if you should find it worthy for your students. It is quite 
> > elementary and intended to be a basic method for finding those spiders 
> > that are easy to see. At any rate use the info contained in any way you 
> > see fit. 
> > 
> > Servo

A Tale of the Spider in a Jar

It is the many spiders of the web that help us to find what we seek to know. They crawl tirelessly about the web while recording their journies in order that we may find those places where they have been. These spiders are many and include those such as Scooter, Slurp, Gulliver, fido and many more. They build the indexes we search at AltaVista, HotBot, Northern Light and elsewhere in our quest for knowledge. While these spiders toll to ease our journey there is much knowledge they never see and thus fail to record. Knowledge hidden amongst the multitudes of worthless articles on the web which pretend to convey knowledge or that knowledge which is hidden in private webs that aren't indexed by those spiders which crawl only on the main web for lack of a route to the knowledge kept in private webs.

This is where the story begins of the spider in a jar. There are actually many of these about the web laying quietly along the edges. These spiders only crawl their own little space and are kept from the larger web of the internet. They crawl the small private webs of the universities and private business making a journal of the knowledge hidden within that is normally not visible or diluted with many irrelevant places making them hard to see. The journals of these spiders contain only records from a small private web or a limited topic from the larger web. Using these journals for your queries can help filter the many worthless sites that may be given by the unrestricted spiders.

What's that you say? Where can we find these spiders? You must search for them in their hiding places. To find them you can use the journals of the free roaming spiders that usually pass over these smaller spiders but record there location. Let's begin with locating a small spider to help us relax while we are searching or working about other tasks, a spider that works to record cocktails :-) While you should already have the recipes on hand for an assortment of helpful cocktails you may wish for something different or if you are a newbie you may need to begin with a place to look. Try searching Scooter's index at AltaVista with the query "cocktail search" or try the link

http://www.altavista.com/cgi-bin/query?pg=q&sc=on&hl=on&q=%22cocktail+search%22&kl=XX&stype=stext

and you will be returned a list of pages which contain search tools which will only look up cocktails :-) They will contain nothing else in their index to dilute your results. In one fell swoop you have filtered all of the irrelavant returns.

Perhaps you do not need a new recipe for a cocktail but wish to find a spider that indexes Legal Resources on the web. We can begin with a search of scooter's journal with +search +legal or +search +law where we will have returned to us some links to spiders which record only pages related to law. With these spiders we can search to find answers to questions on the law that might otherwise be difficult to locate in the diluted results of the free roaming spiders. If you wish to search for pages of medicine try +search +medicine and you'll have returned you some spiders which index only medicine. Quite an elementary task is this to find journals where the knowledge is more pure.

These searches will return mainly spiders which roam freely over the large web will contained within the topic of the jar which contains them. There are spiders though which only crawl on small webs on the inranets of universities. These webs contain a wealth of knowledge which is totally invisible to the free roaming spiders. Perhaps in the example of medicine above you wish only to find the spiders of places like medical schools. Again, quite an elementary task. We add to our search above, +search +medicine domain:edu so that we are returned only the pages that have both of the terms search and medicine in the edu domain. We could do the same for anything we want to find from the schools from art to zen.

Now it is time that you shall work. Go forth and find spiders from art jars, programming jars, wine jars and more. Look over the web with all of the spiders you can find for all the little spiders that are trapped as servants to their small fields of knowledge where the big spiders cannot go. Find the ones that roam the big web and record only the pages for which their jar was made to hold. Among them you will find many which will help you to find the knowledge where only they have gone...