Decided to remove the referrers from the sidebar. I was noticing it seemed to take a very long time to load the page. I figured it was because of the referrer lists, so I added some code for timing to the scripts and sure enough that was it. It was taking 2 to 4 seconds to do the non-search referrer lists. Which made sense because each referrer had to compare to each line of the search engine lists, which currently has 269 lines in it. Could probably make it faster by sorting the list and doing a real search on it, which would lower the number of compares. But I didn’t really want to get into it, so I just moved it out of the sidebar.
I put it in an item, and added a link in the sidebar (at the top after archives) to that item. But then I found out you can’t change the date of an entry to the past. I was going to put the entry back at the beginning of the year so it wouldn’t be on the main page right now. But I couldn’t. So it will have to cycle off like other entries.
Wanted to get some of the search engines that were in the referrer list and not search keyword list over so I added a bunch to my SearchEngineQueryList.txt file.
Its the sign of a good script when you can change the functionality quickly in a data file.
Given that I only have one referrer log file, if you get a lot more of one kind of hit than another then the lesser type will be pushed out completely. I came in this morning and there were no referrer’s listed, but there was a new comment, which indicated to me someone had linked over here.
The simple solution was to bump up $maxref in reflog.php. This will cause more to be saved in the log, but doesn’t change how many are displayed. I now save 400 instead of 100. There is probably a slow down each time reflog.php is executed, but hopefully not too much of one.
I managed to get both the regular referrer and the keyword referrer scripts into one file. But then I had to break them out because of the way PHP includes works under Nucleus. Still most of the functions are in one file and there are two small files for including in my pages.
Here’s a break down to each file:
reflog.php – the file that does all of the logging. It creates the file the others read. It logs a set number of log items. I set this to 100 because it should contain both the search and non-search referrers. It puts the date and time in each line as well.
searchfunctions.php – work horse of the display. It contains functions for doing the actual building of the html from the log file. There are three functions in it.
function displayReferrers($displayType,$referrerFile, $displayLimit)This used to be the body of the one file version. Now it is what is called by the other files. You pass if what type of display you want. “Search” is the only option at the moment, and if you pass in an empty string or anything else you the the standard referrers. You pass in the relative path to the referrer log file, and the max number of items you want displayed.
function killURL($inURL)This function read in a file called “ref_kill.txt”. It then searches for each line in the passed in url. If it finds it, it return YES, meaning don’t display this URL. You put strings on one line each in the text file.
function getSearchEngineAndKeywords($inURL) This routine parses out the query words and returns the search engine if it is a query. If not it return null.
showSearch.php and showReferrers.php – these two do basically the same thing, but with different parameters for the call to displayReferrers.
Update: Found a bug in killURL that was causing it to only filter on the last item in the kill file. You need to trim the search first, or it will include the return character. The online script has been updated.
Update 2:Made a change so if you get a URL from a search engine but it has no query string getSearchEngineAndKeywords will return null instead of an empty query. Did this because I added Technorati to my search engine list. Now queries show up in the query list, while pings show up in referrers.
I did a major rewrite of the search word referrer script I got from scriptygodness.
What this does is read a file that contains URLs, assumably referring URLs, and generates the HTML. I am currently using the lgf script above to generate the file. I’ll be working on one of my own soon.
The original script called a method that gave back the referrer keywords. It used a big array hard coded into the script to do the matching. I hate hardcoded data, so I wanted to get this stuff in to a file. Also it had data that gave you the name of the search engine which I thought was cool and that I’d like to display, but the script didn’t return this information.
So I rewrote the whole thing. I moved the data into its own file and read it in to create the array. Then I use pretty much the same code to figure out the keywords, but return the search engine name along with the keywords.
Here is the script if you want to look at it or use it. Here is a link to the search engine text file.
One thing to notice is I rearranged the search engine list in the file so Google and Yahoo are at the top. I did this because they are the ones I get the most hits from and the search of the array will be done sooner.
Been doing a little work on the site and this has involved writing some PHP code. I thought I’d share the scripts so others could benefit. I’ve created a new category for scripts and moved a couple of previous posts to it.
Today I wrote a script to read in an OPML file and output some HTML to a blogroll. Actually someone else wrote most of the PHP to do this I just modified it a little for Reactuate. Here is the script as it runs on Reactuate.
The reason I wrote the script was I use a news read called NetNewsWire Lite to read blogs. I wanted to post a list of the blogs I read on a regular basis. I thought I’d use AppleScript to read the list and create an HTML file. Well it turns out NetNewsWire Lite doesn’t do AppleScript. I thought I’d be upgrading to NetNewsWire Pro, but after using it during the beta period I decided it wasn’t going to use it for posting so I didn’t really need the full version.
So AppleScript was out, but NNW Lite will output your subscription list as a OPML file. So I did that and then needed something to make HTML out of that. I found the script used here and made some mods to get it look right here and let me put my OPML file in another location.
Now all I have to do is upload new OPML files and the blogroll with change on the fly.
I’ve added some code to the lgf-referrer script I’m using to block certain domains from the list. Mostly I’ve blocked search engines, since I have them in my other list. But I just figures out two new ones to block today, http://radio.xmlstoragesystem.com/ and http://www.technorati.com/.
http://radio.xmlstoragesystem.com/ is the weblogs.com ping checking that my site has changed. I ping weblogs.com whenever I change the site. Sometimes I go there just to find random new blogs, and I figure other people do the same. But I don’t need their robot in my referrer list. I don’t like to know when someone comes from there to my site, so weblogs.com shows up in the list.
Technorati is a cool site, but they hit my site every time I come up on weblogs.com. Unfortunately blocking them means I won’t see when people come from their site to look at the site.
I finally got around to adding a referrer log to the page. I got if from http://www.scriptygoddess.com. The main referrer log is pretty cool. But I find that most of my referrers are from search engines, so I went looking for a solution.
First was a script that lists keywords from search engine referrers. I added it, but then I didn’t want search engine URLs to show up in the main referrer list. I found a solution here.
Now I have two lists. One that should be actual referring links and one that is a list of the keywords that bring people here.
I’m going to clear the lists right now because I’ve been getting them to add referrers for testing and the old search engine URLs are still in the main list.