How to extract images from your squid cache

When I searched for this, I found a lot of questions but no real answers.  Based on a couple of hints I found online, I put these scripts together which might be of use to someone else.

First, the script to scan the cache for images we want to extract.

#!/bin/bash

# -- Edit these as needed !
cache=/var/spool/squid
output=/home/johncc/Out/
filebase=foo-XXXX
# --

match="$1"

set -u

# Limit to files above 15k size in order to skip thumbnails where possible
find "$cache" -size +15k -type f | while read file
do
    if head "$file" | grep -q "$match"; then

        content_type="$(head "$file" | grep -ai "Content-Type:" | cut -f 2 -d " " | tr -d '[:cntrl:]')"
        echo "Matched $file ... content type $content_type" >&2

        type=""
        if [[ "$content_type" = 'image/jpeg' ]]; then
            type=jpg
        elif [[ "$content_type" = 'image/png' ]]; then
            type=png
        elif [[ "$content_type" = 'image/gif' ]]; then
            type=gif
        fi

        if [[ "$type" != "" ]]; then
            outfile="$output$(mktemp $filebase).$type"
            echo "Converting $file as $outfile with type $type" >&2
            squid_get_image -q -t $type $file $outfile < /dev/null
        fi

    fi
done

Save this as ~/bin/squid_scan or similar (if ~/bin is in your path) and do chmod 755 ~/bin/squid_scan

This uses some fairly rudimentary filtering, which could be improved. To use it to grab all images from mysite.com, you call: squid_scan mysite

As you can see, this calls squid_get_image to actually do the image extraction. Here’s the code:-

#!/bin/bash

set -u

usage()
{
    echo "squid_get_image [-q] [-t ] infile outfile" >&2
    echo >&2
}

im_type=jpg
quiet=false

while getopts ":qt:" OPTION
do
    case $OPTION in
    t)
        im_type=$OPTARG
            ;;
    q)
            quiet=true
            ;;
    \?)
        echo "Invalid option: -$OPTARG" >&2
        usage
        exit 1
        ;;
    esac
done

shift $(($OPTIND - 1))

if [[ $# != 2 ]]; then
    echo "Expecting 2 parameters, got $#" >&2
    usage
    exit 1
fi

infile=$1
outfile=$2

$quiet || echo "Coverting $infile to $outfile" >&2
case $im_type in

jpg)

    bvi -f <(echo -e '\FFD8\,$w' $outfile\\nq) "$infile" &>/dev/null
    ;;

png)

    bvi -f <(echo -e '\89504E470D0A1A0A\,$w' $outfile\\nq) "$infile" &>/dev/null
    ;;

gif)

    bvi -f <(echo -e '/GIF8[79]a/,$w' $outfile\\nq) "$infile" &>/dev/null
    ;;

*)

    echo "Bad Image Type." &>2

esac

NOTE: You need the bvi utility (binary vi). In Debian/Ubuntu just sudo apt-get bvi. In Archlinux it’s in aur.

Save this as ~/bin/squid_get_image or similar and do chmod 755 ~/bin/squid_get_image

This uses bvi’s ability to run a script of ex commands. It uses the fact that the squid cache files are some binary data, the HTTP headers, and then the file itself. The gif, jpg and png formats all start with “magic numbers”, and the ex commands look for those before writing out everything that follows to the specified file name.

I hope this is useful to someone!