Crawl a website to get a list of all URLs

Sometimes you just need a list of URLs for tracking. There exist loud & fancy tools to crawl your site, but you can also just use wget. wget --mirror --delete-after --no-directories http://your.website.com 2>&1 | grep '^--' | awk '{print $3}' | sort >urls.txt or, if you just want the URL paths (instead of http://example.com/path/filename.htm just /path/filename.htm): wget --mirror --delete-after --no-directories http://your.website.com 2>&1 | grep '^--' | awk '{print $3}' | sort | sed 's/^.

Crawl a website to get a list of all URLs »

Command lines for the weirdest things

Just a collection of command line tweaks. These work on Linux, probably mostly on MacOS too (and who knows, with things for Windows probably). (content irrelevant, just adding an image) Basics General command-line tips Todo: find some other sources Pipe tricks: | more - show output paginated (use space to get to next page) | sort - sort the output lines alphabetically | uniq - only unique lines (needs “sort” first) | uniq -c - only unique lines + include the number of times each line was found (needs sort first) | sort -nr - sort numerically (“n”) and reverse order (“r”, highest number first) | wc -l - count number of lines Searching for things (more grep options)

Command lines for the weirdest things »

CO2 in your meeting room / office

Want to know what level of CO2 you have in your meeting room or office-space? Want to know when you need to start ventilating air to stay productive? Here’s a simple calculator that works out how much CO2 people produce (when they’re not overly active), and what level of CO2 that produces in a closed space. Simple calculator Room size [in meters]: (wide) x (long) x (high) Number of people: recalculate

CO2 in your meeting room / office »

Tiny USB keyboard with ATMEGA 32u4 - it works!

After a USB keyboard with an ATTINY85, a try at one with ATMEGA 32u4, I’m now at revision 2/3 for the ATMEGA 32u4 single key, USB keyboard. Overview Same as before: a simple USB keyboard with 1 key reprogrammable tiny mechanical keyboard key cheap enough to give away actually works debuggable The “cheap” aspect was mostly to justify making & buying some :). Hardware design Since the previous design mostly worked, I tweaked to make two versions.

Tiny USB keyboard with ATMEGA 32u4 - it works! »

Crawling all (most) of the web's robots.txt comments

Starting from this tweet … View tweet … I hacked together a few-lined robots.txt comment parser. I thought it was fun enough to drop here. Crawling the web for all robots.txt file comments curl -s -N http://s3.amazonaws.com/alexa-static/top-1m.csv.zip \ >top1m.zip && unzip -p top1m.zip >top1m.csv while read li; do d=$(echo $li|cut -f2 -d,) curl -Ls -m10 $d/robots.txt | grep "^#" | sed "s/^/$d: /" | tee -a allrobots.txt done < top1m.

Crawling all (most) of the web's robots.txt comments »

Redirecting from Blogger with JavaScript

While it’s possible to redirect a whole site from Blogger to the exact same URLs, redirecting to similar URLs is hard. Let’s say you want to redirect: from: https://yoursite.blogspot.com/2012/01/blogpost-name.html to: https://newsite.com/blogpost-name The way to do this is to drop a snippet of JavaScript into your Blogger site’s template. Adding a redirect like this also makes it impossible for you to view your blog, so be careful :-). Also, adding a redirect like this means you won’t be able to edit your site’s theme using the simple tools (you’ll need to edit the HTML) - usually that’s no big deal since you want the old site to disappear, right?

Redirecting from Blogger with JavaScript »

Tiny (still-flakey) USB keyboard with ATMEGA 32u4

After making a USB keyboard with an ATTINY85 and noticing it wasn’t the “yellow of the egg”, I decided to try at making an ATMEGA 32u4 version. Overview What I was looking to make was (recap from previously): a simple USB keyboard with 1 key reprogrammable tiny mechanical keyboard key cheap enough to give away actually works debuggable The “cheap” aspect was mostly to justify making & buying some :).

Tiny (still-flakey) USB keyboard with ATMEGA 32u4 »

Coronavirus / Covid-19 notifications could not be activated: FAILED_SERVICE_DISABLED

Got a new phone and wanted to set up a local Coronavirus / Covid-19 exposure app but it just can’t be turned on? The solutions are simple: either wait a few hours for everything to update, or wait a few years, for developers to make usable error codes. Grr I installed SwissCovid on a new Android phone and tried to activate proximity tracing. It threw a useless error message with:

Coronavirus / Covid-19 notifications could not be activated: FAILED_SERVICE_DISABLED »

Wifi-control for a 5050 LED strip with ESP32

The 5050-type LED strips are pretty cheap, and can (sometimes) be controlled by ESP32’s fairly easily. The rough idea is to connect the individual color pins to a NPN transistor and to control them with an ESP32 PWM output pin. 5050 LED strips are always set to the same color, you can’t control them individually. There are different voltages of strips available - we’re using 5V since they seem to be pretty cheap, and they can just be powered by a USB cable.

Wifi-control for a 5050 LED strip with ESP32 »

Xiaomi Mijia thermometer & hygrometer in Home Assistant (LYWSD03MMC) on Hass.io

Connecting a thermometer & hygrometer (humiditymeter?) to Home Assistant can be a bit of a hassle. Who doesn’t want one in every room? There are lots of ways to get them set up, but it’s easy to make things nicely-complicated & expensive. Who doesn’t love spending more money than you need? I picked up a pack of Xiaomi Mijia sensors (or LYWSD03MMC, when you get to know them), they run on a coin-cell battery for about a year, have BTLE / Bluetooth Low Energy transmission of their status, and are pretty cheap.

Xiaomi Mijia thermometer & hygrometer in Home Assistant (LYWSD03MMC) on Hass.io »