This is pretty much cross-platform, but as a Windows-user you’ll have to grab and install two files first:

• wget – a tool to download copies of web pages
• UnxTools – a collection of popular Unix/Linux tools for the hacker in you

Extract the ZIP files, copy the contents somewhere where you can find it and make sure that the appropriate folders are in your “path” (the files you’ll need for UnxTools are in “…\usr\local\wbin”). We’ll need to access these tools through the command line. I have a feeling I may need to elaborate on that for Windows users — let me know if that’s the case.

First, we’ll mirror our site on our local machine (this assumes that your site is crawlable; if it isn’t, then fix it first ):

1. Open a command box or terminal window (on Windows, hit Start / Run … and enter “cmd”)
2. Go to or create a temporary folder
3. Run the following command to mirror your site:

   wget --mirror --accept=html,htm,php,asp,aspx http://domain.com/

   This command mirrors pages with .html, .htm, .php, .asp and .aspx extensions on http://domain.com/. It’ll create a folder for the domain and put all the files in it. Dynamic URLs will get adjusted so that they can be used as file names.

4. Wait … until it’s all downloaded … if it feels endless, you might have endless URLs, perhaps an infinite calendar script or something similar? It’s worth fixing!

Alrighty, now that we have a copy of your site, let’s check things out.

Finding pages without Analytics

We can find pages without the Analytics tracking code by listing all pages which do not have certain content in them:

grep -r -L "google-analytics.com" *.*

This command goes through all subfolders (the “-r” option) and lists the files that do not contain a match (“-L”) for “google-analytics.com”. That could be extended to just about anything .

How about pages that don’t have a “description” meta tag?

grep -r -L "meta name=.description" *.*

The “.” (period) matches any character — in this case, it is used to match the ” (double-quote).

Finding pages with AdSense (and the ID used)

Finding pages that contain a certain text is even easier:

grep -r "google_ad_client" *.*

Note that all we did was drop the “-L” (and change the text, obviously). It will show the lines that match this pattern in all of your pages, which includes the AdSense ID.

Similar to the earlier check for missing “description” meta tags, assuming you have the contents of that tag all in one line, you can easily find all of these meta tags with:

grep -r "meta name=.description" *.*

What would you like to search for today?

I’ve been trying to do that for years and did the most exotic things to make it happen. I’ve used four different browsers in parallel and I’ve even used a virtual PC running within my PC (that kind of defeats the desire to use less memory, but it feels neat anyway). In the end, a collegue in the office, who happens to use emacs as his main web browser , pointed me into the right direction.

Now I have three completely independant instances of Firefox running at the same time!

So what’s the trick?

Firefox has command line options to let you start multiple profiles and specify a certain one. In our case, we’re going to change the command line to:

“C:\Program Files\Mozilla Firefox\firefox.exe” -no-remote -P NewProfileName

To get started, check the name of your current profile. On Windows you can find it in “c:\Documents and Settings\[user-name]\Application Data\Mozilla\Firefox\Profiles”. It will generally have a few characters and numbers, a period and then the profile name (in my case it was something like “36fc232a.default”). Use this to adjust the settings of the icon you use to start up Firefox. On Windows, right-click on the icon and select “Properties”; you can add the options in the field called “Target”:

If you click on that icon now, it should start up Firefox just as before (ok, this is not the neat part yet ).

Now make a copy of the icon (I right-click drag it into a folder and select “Copy”) and change the command line options (and file name) again, only this time choose a different profile name. If you want to use a copy of your existing profile (with all cookies, bookmarks, themes and add-ons), you can do that by going into the folder where your profiles are stored (mentioned above) and copying your default profile. Now when you start up Firefox with that icon, it will bring the profile manager since it can’t find that new profile. Create a new profile and use the exact name you used in the options. You will then have a choice of either creating a completely new profile or using an existing profile folder.

Now you have two instances of Firefox running at the same time. They’re completely separate, so if one crashes, the other will continue normally. If one starts using too much memory, you can close it and restart it without impacting the other one. If you have conflicts with add-ons or want to use different cookie sets, just use a separate instance.

Since the various instances will generally look the same and be hard to keep apart, I just applied different themes to them. The “Safari-style” theme is my private one, the blue one is used for all my work-apps and the normal one is used for all kinds of testing.

This trick should work on all platforms with Firefox, not that I tried it out so try it at your own risk . Now if only I could migrate my IE profile back to Firefox …

What we’ll do is “sniff” the connection that your FTP client builds up, we’ll take a look at the packets sent out and received. Remember that other people can do this as well – say if you’re on an insecure wireless connection on the road — use secure connections and protocols whenever you can!

You’ll have to get a copy of Ethereal (freeware), a universal network analysis tool (there are many similar tools available, I like the flexibility of Ethereal). Download it, install it and start it up.

To get started, select the menu item Capture and Start, then choose your ethernet interface (WLAN, cable, etc) and let it start. You are now recording your complete network traffic, you 1337 self-h4×0r . Depending on what you’re doing at the moment, it may record a lot of traffic. We’ll filter it later on, so don’t worry about that.

Now start up your FTP client (make sure you’re not using a secure FTP connection) and connect to your server. When you connect to your server like that, you will send your user name and password over the network and Ethereal record that for you. Once you have that, you can stop capturing in Ethereal.

If you scroll through the data you collect like that, you’ll quickly notice that there’s a lot going over those wires. Let’s just look at the data going to and from our FTP server. You’ll have to get the IP address of your server (which you can usually do in a shell/command box by typing “nslookup ftp.yourservername.com”). In the filter box on top, enter: ip.addr eq nnn.nnn.nnn.nnn (where the “nnn’s” are the IP address of your server).

Once you only look at the data going to and from your server, you’ll see the authentication information right away:

Now that you see how easy it is to hack yourself, make sure that others can’t do the same with your account:

• If you’re using a wireless connection, always assume that others can listen in (even if you’re using your own access point with WEP or WPA encryption).
• Make sure that you use a secure version of FTP. In general, they will encrypt your authentication information so that it will not be readable on your network. Double-check it with Ethereal, if you want to be sure.
• Change your FTP/email passwords after you have used them on an insecure connection like a hotel or airport wireless.
• If you use a web-based email service, make sure that you are accessing the site with HTTPS and not HTTP. Most web-mail services allow that (though they may not activate it by default since it is a bit slower and is usually not needed on your home network).
• Even if your FTP (or email) client encrypts passwords in the settings, they can still be read with the right tools.

Stay safe!

Python is a neat little programming language, I like it more and more as I use it .

Here’s how to get started:

1. If you do not have Python installed, go and download and install Python. I assume most Apple OSX will have it installed, but I don’t have a Mac so I can’t say for sure. It’ll almost certainly be installed if you’re one of the 3 Linux-users who have visited my blog. If you’re using Windows, take the version with the installer (it’s easier) and make sure that the folder where you installed Python is in your “path”.
2. Grab my wmtextract01.zip and extract it into a folder. You should have three files: wmtextract.py (the Python script) and ProcessAll.bat + ProcessAll.py.
3. Copy your query stats CSV files into that folder as well.
4. Double-click on ProcessAll.py (or ProcessAll.bat if you’re on Windows and don’t have Python set up to run scripts directly)
5. The script will now process all CSV files in the same folder, create a new folder called “output” and place the new CSV files there.
6. Open the new CSV files in Excel (or Open Office or even Google Docs + Spreadsheets)
7. Enjoy

Here are some more ideas for the CSV files when you have them in Excel:

• Select everything (Ctrl-A) and set up an “AutoFilter” (menu item Data / Filter / AutoFilter). Now you can filter your stats however you want them. Want to only see the queries for which you rank #1? How about the queries that people from Switzerland used to find your site?

• Set the Location to “All locations” and search type to “All searches”, now select everything and sort by the last column (menu Data / Sort / has header row / sort by Column E, ascending). Now select the last two columns (D and E) and click on the chart icon. Choose an “XY scatter” chart and let it create it. This chart shows you the ranking of your site for the search queries. There are some problems with the chart like this (keywords can be listed several times), but I think it’s neat anyway

What’s the neatest information you ever found in your webmaster tools query stats?

The tools I included in this set can do the following:

• Check the result codes for a URL (and follow in the case of a redirect) – or for a list of URLs
• Create a list of the links found on a URL (or just particular ones)
• Create a list of the links and anchor texts found on a URL (or just particular ones)
• Create a simple keyword analysis of the indexable content on a URL

You can get the down from here (requires the Windows .NET runtime v1.1):

• WebToolbox.zip (140kb)

WebResult

This tool accesses a URL and shows the result code that was returned. If the status is a redirect, it will display the redirection location and optionally follow it to check the final result code. It may be used with a list of URLs. The output is tab-delimited.

Usage:
WebResult [options] (URL|urllist.txt)
Options:
–referer|-r [referrer] (default: none)
–user-agent|-u [user-agent] (default: “WebResult”)
–follow-redirect|-f (default: not)
–headers|-h (displays the full response headers)
–verbose|-v

Example:
Check for correct canonical redirect:
Webresult http://johnmu.com/
Webresult http://www.johnmu.com/

WebLinks

This tool lists the links that are found on a URL. Note that it has an integrated HTML/XHTML parser – if the code on the page is not fully compliant, there is a chance of the parser not recognizing all links (it is fairly fail-safe, though).

This tool can use a cached version of the URL (from either this tool or one of the other ones) to save bandwidth. The cached versions are saved in the user’s temp-folder.

You have the choice of only listing domain outbound or insite links (to help simplify the output). Additionally links with the HTML microformat “rel=nofollow” may be marked as such. The output is in alphabetical order.

Usage:
WebLinks [options] (URL|urllist.txt)
Options:
–referer [referrer] (default: none)
–user-agent [user-agent] (default: “WebLinks”
–insite-only|-i (default: both in + out)
–outbound-only|-o (default: both in + out)
–ignore-nofollow|-n (default: off)
–cache|-c (default: off)
–verbose|-v (default: off)

Example:
Check the outbound links on a site.
WebLinks -o http://johnmu.com/

WebAnchors

This tool lists the links and anchor text as found on a URL. It uses the same HTML/XHTML parser as WebLinks. It can be used to find certain links (based on the URL, domain name, URL-snippets, or even parts of the anchor text). If the anchor for a link is an image, it will use the appropriate ALT-text, etc.

Usage:
WebAnchors [options] (URL|urllist.txt)
Options:
–referer|-r [referrer] (default: none)
–user-agent|-u [user-agent] (default: “WebLinks”
–find-url|-f http://URL
–find-domain|-d DOMAIN.TLD
–find-anchor|-a TEXT
–find-url-snippet|-s TEXT
–url-only|-o (default: show anchor text as well)
–skip-nofollow|-n (default: off)
–cache|-c (default: off)
–verbose|-v (default: off)

Example:
Check the links with “Google” in the anchor text.
WebAnchors -s “Google” http://johnmu.com/

WebKeywords

This tool does a simple keyword analysis on the indexable content of a URL. It also uses the above HTML/XHTML parser to extract the indexable text. It is possible to get single-word keywords or to use multi-word-phrases. The output is tab-delimited for re-use.

Usage:
WebKeywords [options] (URL|urllist.txt)
Options:
–referer|-r [referrer] (default: none)
–user-agent|-u [user-agent] (default: “WebLinks”
–verbose|-v (default: off)
–words|-w [NUM] (phrases with number of words, default: 1)
–ignore-numbers|-n (default: off)
–cache|-c (cache web page, default: off)

Example:
Extract 3-word keyphrases from a page:
Webkeywords -w 3 http://johnmu.com/

Combined usage of these tools

Find common keyphrases on sites linked from a page (uses a temporary file to store the URLs):

webanchors -c -o -a “Google” http://johnmu.com >temp.txt
webkeywords -c -w 3 temp.txt

Check result codes of all URLs linked from a page:

weblinks -c http://johnmu.com >temp.txt
webresult temp.txt >links.tsv

Compare result codes for multiple accesses:

echo. >results.tsv
for /L %i IN (1,1,100) DO webresult http://johnmu.com/ >>results.tsv

or more complicated to test a hack based on the referrer (all on one line):

for /L %i IN (1,1,100) DO webresult -u “Mozilla/5.0 (Windows; U) Gecko/20070725 Firefox/2.0.0.6″ -r http://www.google.com/search?q=johnmu http://johnmu.com/ >>results.tsv

I’d love to hear about your usage of these tools .

There are services available that can track changes on your web pages automatically, but sometimes it is good to have something like that within your own control (or perhaps as a backup to an online service). To keep a record of changes on web pages I have put together a small Windows batch-file that checks a list of pages and emails you with any changes found. Additionally, it will also email you when the server is not reachable. You could use the same tool to keep track of changes on third-party web pages.

The download is available here:

• WebCheck Batch file v0.1 – 4kb / 18. Aug 2007

In order to use this tool, you will need two additional downloads:

• WGET (to download web pages): you can get the Windows program files at http://xoomer.alice.it/hherold/
• BLAT (to send emails): you can download this at http://sourceforge.net/project/showfiles.php?group_id=81910

Follow the instructions in the “readme.txt” within the ZIP file to install and set up this script.

Note: As mentioned in the instructions, you cannot list URLs with parameters directly – you need to use tinyurl.com to create a short version, which can be listed.

Once you have set the files up, make sure that the “checkurls.txt” file has the URLs that you want to track (as well as a short identifier) and then just doubleclick “checkall.bat”. You could also use the windows scheduler to automatically start that URL, or put a shortcut to it into your autostart folder to have it started whenever you log in.

One URL to test it with is http://johannesmueller.com/ – within that page the server embeds a counter as a HTML comment. The program should automatically signal that URL every time you start the program. If you include a URL like that within your list of URLs, you can be fairly certain that the program is working properly as long as you receive a notification for that URL.

The email sent to your account contains a listing of all changes (with line numbers) based on the windows tool “fc” (file compare).

The code (batch file) is released into public domain – but I would really appreciate a short notification of any changes that you might have done. Yeah, I know, batch files are sooo 80’s . It would be trivial (except perhaps for the file comparison) to convert this tool into something that runs standalone, but as a batch file almost anyone can modify it as they see fit, without any fancy programming environments installed.

Adding this kind of markup to your site is pretty easy to do.

1. Download the script and the nofollow image.
2. Open the script file in a text editor (Notepad is fine) and adjust the settings on top:
   Select code
   JAVASCRIPT:

   1. // settings, modify as desired
   2. var myInfoPage ="http://domain.com/nofollow-info.htm";        // local page with nofollow information
   3. var myNofollowImg = "http://domain.com/images/nofollow.gif";        // source of nofollow image (check for full path!)
   4. // warning must be html-formatted
   5. var myWarnMsg = "Beware: this link is potentially bad or not trusted! Click here to find out more.";

3. Upload the script and the image (place the image in the path you specified in your script), don't forget to create your informational page for the nofollow (I'm just linking to my nofollow-policy page).
4. In your template, right before the closing "</body>" tag, add a reference to the script. In a Wordpress theme you would add this to your footer.php, eg:
   Select code
   HTML:

   1. <!-- nofollow highlighting -->
   2. <script src="<?php bloginfo('stylesheet_directory'); ?>/highlight-nofollow.js" type="text/javascript"></script>
   3. ... </body> ...

5. Refresh your browser and check the results. I have a local test page which shows it in action (and here's a nofollow'ed link thrown in for good measure).

Currently the code only adds the image and underline to links that contain a text anchor. This means that links with only an image as the anchor are not modified (but the CSS nofollow highlighting will still be displayed, so be careful if you use that and have image-only links with rel=nofollow). The code was tested on IE 6 and 7.0, Opera 9.01 and Firefox 2.0.0.5.

The code removes the rel=nofollow from the displayed page (it still remains in the static page) so that the different kinds of nofollow markup (including those you might have installed in your browser or with a browser plugin) do not create conflicts. Just to be sure: this script does not change anything on your actual pages, it only changes how the page looks to the user. A search engine crawler would not see these changes.

Advanced users can modify the markup that is added to each link, perhaps even use it to open a floating window on top of the current page instead of going to a separate page.

Current version is v1.0 - 2007-07-28

That said, I believe that all users should be able to recognize these links as being problematic (or untrusted) at first glance. The more advanced, web-savvy users will sometimes have tools installed or special settings which highlight links with "rel=nofollow", the normal user does not. If a link is marked as being problematic / untrusted for search engines, the user should see that as well -- it's nothing more than the Google Webmaster Guideline "Don't deceive your users or present different content to search engines than you display to users (...)". It is deceiving to show users a normal link and at the same time tell search engines that you really don't want to link there! In my opinion, if the link is not good enough, the user should be informed. There are two ways to do that.

My favorite solution would be for browsers to automatically highlight these kinds of links on all sites. However, I doubt that will happen any time soon...

The other solution would be for websites to highlight their own rel=nofollow'ed links. I am doing that here on this site (for the major, modern browsers at least) - an example can be found in my linking policy. Links with the rel=nofollow microformat are underlined and displayed with a small warning symbol.

Adding that to your own site is easy enough - it just requires a small change in your CSS stylesheet and uploading the symbol to your server.

1. Download the symbol: (right-click on it and save a local copy of it). Copy it onto your server into the folder where you have your stylesheet file (this depends on your site - on Wordpress it will be in your theme folder)
2. Open your stylesheet file and add the following lines to the end of the file:
   Select code
   CSS:

   1. a[rel~=nofollow]
   2. {
   3.     border-bottom: dashed 2px #ffa500;
   4.     background: url('nofollow.gif') no-repeat center left;
   5.     padding-left: 17px;
   6. }

   If you have placed the image in a different folder, you need to specify it here. If you with to use a different sized image, make sure you adjust the "padding-left" setting (this makes room for the symbol). If you have an orange background on your site, also make sure that you adjust the color from "#ffa500" to something that contrasts the background (in that case you might also consider changing the colors in the symbol). Upload the stylesheet to your server.

3. Done!

   You should see the changes when you open a new page on the site. In some cases you might have to restart your browser to see the modified stylesheet.

Do you think that a global highlighting like this would make some sites take a second look at their policy of adding "rel=nofollow" to all outbound links? Imagine you had an encyclopedia where all references were highlighted as being untrusted .

1. Make a backup

   Before you change anything, always make a backup of your theme files and perhaps even of your full Wordpress installation. It never hurts to have too many backups .

2. Download the script

   Download the file textresize.js from here. Put it on your desktop or someplace where you can find it.

3. Check your Wordpress theme

   You might be able to skip this step, depending on how your Wordpress theme is made. Open your blog in your browser, use "View Source" and take a quick look at the source code (not too long, it could make your head spin). You should find a section pretty much on top which "wraps" your whole site in a "DIV" element. Often it will be called "wrapper", "content" or "container". It will look something like this:

   Select code
   HTML:

   1. <div id="wrapper">

   If you find multiple names that you believe could be the main elements, you might want to try them separately to see which one works best. The default in the code is "wrapper".

4. Update the script to match your theme

   If your theme does not use "wrapper" as the ID of the main element, you will need to adjust the javascript file. Don't worry, it's not painful . Open your copy of "textresize.js" in a text editor like Notepad. You will find the following line near the top:

   Select code
   JAVASCRIPT:

   1. var ts_styleElement="#wrapper"; // the ID of the content DIV that you want to resize

   Change that line to match your main element, eg:

   Select code
   JAVASCRIPT:

   1. var ts_styleElement="#content"; // the ID of the content DIV that you want to resize

   Save the file before a cat walks across your keyboard.

5. Upload the script to your server

   Copy the modified file onto your server into the current theme folder.

6. Update your theme files

   Add the following to your header.php in the bottom of the "head" section:

   Select code
   HTML:

   1. <script
   2.    src="<?php bloginfo('stylesheet_directory'); ?>/textresize.js"
   3.    type="text/javascript"></script>

   Now add the following wherever you wish to see the resize links:

   Select code
   HTML:

   1. <script type="text/javascript"><!--
   2.     showReadabilityMenu('Resize text:');
   3.     //--></script>

   I have placed this in my header.php, right after the "searchform" include.

7. Optionally add more styling

   You can style the line with changes to your CSS file, eg:

   Select code
   CSS:

   1. #header #fontsizemenu
   2. {
   3.     padding-top: 2px;
   4. }

   
   This will depend on your theme and on what you want to do to integrate the links.

8. Test, test, test

   Be sure to test the new functionality in the major browsers. I have tested the code in IE 6.5, IE 7.0, Firefox 2.0 and Opera 9.0. The code depends on javascript. With javascript turned off, the user will not see anything.

Have fun!