regexp - jwz

fail2ban GC

jwz — Thu, 08 Jan 2026 08:30:44 +0000

Over the years I've accumulated thousands of fail2ban regexps. It's a performance impact. Is there some way to figure out which ones have not been hit in a while, to remove?

Previously, previously, previously.

Today in "Now you have two problems" news:

jwz — Tue, 27 May 2025 19:37:32 +0000

If you want a picture of the future, imagine ~~a boot stamping on a human face forever~~ a spicy autocompleter submitting software changes to a regular expression parser.

Also, you can't block it. As Kyle Reese reminded us, "It absolutely will not stop -- ever -- until you accept its pull req̷̩̓u̸̾͜ḙ̶̈s̶̫͑t̶̙͒."

Previously, previously, previously, previously, previously, previously, previously, previously, previously, previously.

Now I have two problems

jwz — Mon, 06 Jan 2025 21:12:52 +0000

Lazyweb, why doesn't this regexp match in Perl 5.32.1, 5.34.3 or 5.18.2 but does in PHP and JS:

 perl -le 'print 0 + ("ABCDE" =~ m/ABCF|BCDE|C(G)/);'

Changing nearly any character, including the C before the G, or removing the parens, makes it match.

Previously, previously, previously, previously.

Mastodon stampede returns

jwz — Wed, 30 Oct 2024 00:18:54 +0000

If you've noticed that the Mastodon stampede DDoS is back, hammering your servers even though you have applied countermeasures -- that's because they CHANGED THEIR FUCKING USER AGENT for no good reason and without telling anybody.

2024-07-31 23:39:18: http.rb/5.2.0 (Mastodon/4.3.0-nightly.2024-07-25; +https://mastodon.social/) 2024-08-01 03:41:26: Mastodon/4.3.0-nightly.2024-08-01 (http.rb/5.2.0; +https://mastodon.social/)

Ok, yes, technically the plans were on display in a toilet with a sign on the door saying "Beware of the Leopard".

Could anyone have possibly have predicted that this would be a problem? I guess we may never know.

Prepend "^Mastodon\/|" to your regexps.

Sigh.

Previously, previously, previously.

WebCollage is 25 years old today

jwz — Tue, 18 Jun 2024 19:49:42 +0000

WebCollage feeds random noise into image search engines to create collages, and it has been running continuously on my site since 1999. That's... a lot of image scraping!

These days it's probably not the most interesting web toy, but I'm kind of shocked that it still works. I only give it a little bit of attention maybe every six months, where I tweak URLs and regexps to track the latest obfuscated nonsense that the various search engines spit out.

Over the years of maintenance, WebCollage has given me some unwwanted insights into the "SEO" industry, almost from that vile industry's inception. When the same image keeps showing up repeatedly, it is a strong indicator that some site has managed to poison search results. Back in 1999, the most common technique was for the tag to contain thousands of random words from a dictionary. Early adopters of this technique were porn and genealogy sites. How we have grown.

The very first version of WebCollage polled only Alta Vista and Yahoo! Six months later, I added Hot Bot. Google didn't show up until 2001, Lycos in 2002, LiveJournal in 2004, Flickr in 2005, Twitter in 2008, Bing and Instagram in 2013, and then Tumblr and Imgur in 2017.

In 2002, I got an actual email from a Google engineer asking me to stop searching them, because it was a significant enough portion of their traffic that it was annoying them.

Previously, previously, previously.

Posting this so that people will stop messaging me the "Now you have two problems" joke

jwz — Wed, 10 Apr 2024 17:59:40 +0000

Nazi hellsite does not understand regexps:

If a user typed in "Twitter.com," they would see "Twitter.com" as they typed it before hitting "Post." But, after submitting, the platform would show "X.com" in its place on the X for iOS app, without the user's permission, for everyone viewing the post.
And shortly after this revelation, it became clear that there was another big issue: X was changing anything ending in "Twitter.com" to "X.com."

So you register your phishing domain netflitwitter dot com, post a link to that, and the preview text makes it look like it's going to netflix dot com instead.

Cool, cool.

Previously, previously, previously, previously, previously, previously, previously, previously, previously, previously.

strftime

jwz — Wed, 19 Jul 2023 16:52:02 +0000

PHP 8.1.0 deprecated strftime and related POSIX-based APIs. Their justification, paraphrasing:

Windows screwed the pooch with their buggy implementation of this in the OS, so instead of implementing a POSIX-compatible version in PHP for Windows, we're just going to throw up our hands and tell you not to use it at all.
By the way, of the 38 letter-codes that are used by 'strftime', only SIX of those have the same semantics in PHP's 'date' API, so have fun with that!

So I wrote this to translate from 'strftime' syntax to 'date' syntax:

# php strftime2date.php '%d/%b/%Y:%H:%M:%S %z' strftime('%d/%b/%Y:%H:%M:%S %z') => '03/Feb/2023:04:05:06 -0800' date('d/M/Y:H:i:s O') => '03/Feb/2023:04:05:06 -0800'

Previously, previously.

Cursed Computer Iceberg

jwz — Sun, 11 Apr 2021 20:17:13 +0000

Previously, previously, previously, previously, previously, previously, previously, previously, previously.

now-you-have-two-problems.el

jwz — Mon, 27 Aug 2018 03:25:07 +0000

"I sure am tired of typing extra backslashes in my isearches", I says to myself. "I wonder if anyone has done the unthinkable and implemented perlre in elisp?"

Yes:

(rxt-pcre-to-elisp "(abc|def)\\w+\\d+")

;; => "\$\\(?:abc\\|def\$\\)[_[:alnum:]]+[[:digit:]]+"

pcre2el.el:

PCRE has a complicated syntax and semantics, only some of which can be translated into Elisp. The following subset of PCRE should be correctly parsed and converted:

parenthesis grouping ( .. ), including shy matches (?: ... )
backreferences (various syntaxes), but only up to 9 per expression
alternation |
greedy and non-greedy quantifiers *, *?, +, +?, ? and ?? (all of which are the same in Elisp as in PCRE)
numerical quantifiers {M,N}
beginning/end of string \A, \Z
string quoting \Q .. \E
word boundaries \b, \B (these are the same in Elisp)
single character escapes \a, \c, \e, \f, \n, \r, \t, \x, and \octal digits (but see below about non-ASCII characters)
character classes [...] including Posix escapes
character classes \d, \D, \h, \H, \s, \S, \v, \V both within character class brackets and outside
word and non-word characters \w and \W (Emacs has the same syntax, but its meaning is different)
s (single line) and x (extended syntax) flags, in regexp literals, or set within the expression via (?xs-xs) or (?xs-xs: .... ) syntax
comments (?# ... )
Most of the more esoteric PCRE features can't really be supported by simple translation to Elisp regexps. These include the different lookaround assertions, conditionals, and the "backtracking control verbs" (* ...)

I am sad to report, however, that I can't get it to work in either xemacs 21.5.28 or emacs 22.1.1. (And MacPorts won't let you install emacs and xemacs simultaneously! How partisan!)

Amusing as it would have been had they tried to translate from one regexp syntax to another using regexps... they did not do that.

Also, my heart grew three sizes when I saw that the first line of the file contains -*- lexical-binding: t -*-

Previously, previously, previously, previously, previously, previously, previously, previously.

Just gonna leave this regexp here

jwz — Mon, 16 Jul 2018 19:39:50 +0000

How to handle emoji:

Where other methods are not available, you can use the following regex (for Unicode 11.0 emoji). For clarity, it escapes all characters that can be invisible or are non-spacing -- otherwise you see some odd constructions like ([♀♂])?+ that are really (\\x{200D}[♀♂]\\x{FE0F})?+.
([©®‼⁉™ℹ↔-↙↩-↪⌨⏏⏭-⏯⏱-⏲⏸-⏺Ⓜ▪-▫▶◀◻-◼☀-☄☎☑☘☠☢-☣☦☪☮-☯☸-☺♀♂♟-♠♣♥-♦♨♻♾⚒⚔-⚗⚙⚛-⚜⚠⚰-⚱⛈⛏⛑⛓⛩⛰-⛱⛴⛷-⛸✂✈-✉✏✒✔✖✝✡✳-✴❄❇❣-❤➡⤴-⤵⬅-⬇〰〽㊗㊙🅰-🅱🅾-🅿🈂🈷🌡🌤-🌬🌶🍽🎖-🎗🎙-🎛🎞-🎟🏍-🏎🏔-🏟🏵🏷🐿📽🕉-🕊🕯-🕰🕳🕶-🕹🖇🖊-🖍🖥🖨🖱-🖲🖼🗂-🗄🗑-🗓🗜-🗞🗡🗣🗨🗯🗳🗺🛋🛍-🛏🛠-🛥🛩🛰🛳]\\x{FE0F}|[☝✌-✍🕴🖐][\\x{FE0F}🏻-🏿]|[✊-✋🎅🏂🏇👂-👃👆-👐👦-👧👰👲👴-👶👸👼💃💅💪🕺🖕-🖖🙌🙏🛀🛌🤘-🤜🤞-🤟🤰-🤶🦵-🦶🧑-🧕]([🏻-🏿])?+|🇦[🇨-🇬🇮🇱-🇲🇴🇶-🇺🇼-🇽🇿]|🇧[🇦-🇧🇩-🇯🇱-🇴🇶-🇹🇻-🇼🇾-🇿]|🇨[🇦🇨-🇩🇫-🇮🇰-🇵🇷🇺-🇿]|🇩[🇪🇬🇯-🇰🇲🇴🇿]|🇪[🇦🇨🇪🇬-🇭🇷-🇺]|🇫[🇮-🇰🇲🇴🇷]|🇬[🇦-🇧🇩-🇮🇱-🇳🇵-🇺🇼🇾]|🇭[🇰🇲-🇳🇷🇹-🇺]|🇮[🇨-🇪🇱-🇴🇶-🇹]|🇯[🇪🇲🇴-🇵]|🇰[🇪🇬-🇮🇲-🇳🇵🇷🇼🇾-🇿]|🇱[🇦-🇨🇮🇰🇷-🇻🇾]|🇲[🇦🇨-🇭🇰-🇿]|🇳[🇦🇨🇪-🇬🇮🇱🇴-🇵🇷🇺🇿]|🇴🇲|🇵[🇦🇪-🇭🇰-🇳🇷-🇹🇼🇾]|🇶🇦|🇷[🇪🇴🇸🇺🇼]|🇸[🇦-🇪🇬-🇴🇷-🇹🇻🇽-🇿]|🇹[🇦🇨-🇩🇫-🇭🇯-🇴🇷🇹🇻-🇼🇿]|🇺[🇦🇬🇲-🇳🇸🇾-🇿]|🇻[🇦🇨🇪🇬🇮🇳🇺]|🇼[🇫🇸]|🇽🇰|🇾[🇪🇹]|🇿[🇦🇲🇼]|[\\#\\*0-9]\\x{FE0F}\\x{20E3}|🏳\\x{FE0F}(\\x{200D}🌈)?+|[👯🤼🧞-🧟](\\x{200D}[♀♂]\\x{FE0F})?+|[⛹🏋-🏌🕵][\\x{FE0F}🏻-🏿](\\x{200D}[♀♂]\\x{FE0F})?+|[🏃-🏄🏊👮👱👳👷💁-💂💆-💇🙅-🙇🙋🙍-🙎🚣🚴-🚶🤦🤷-🤹🤽-🤾🦸-🦹🧖-🧝]((\\x{200D}[♀♂]\\x{FE0F}|[🏻-🏿](\\x{200D}[♀♂]\\x{FE0F})?+))?+|👁\\x{FE0F}(\\x{200D}🗨\\x{FE0F})?+|🏴((\\x{200D}☠\\x{FE0F}|\\x{E0067}\\x{E0062}((\\x{E0065}\\x{E006E}\\x{E0067}\\x{E007F}|\\x{E0073}\\x{E0063}\\x{E0074}\\x{E007F}|\\x{E0077}\\x{E006C}\\x{E0073}\\x{E007F}))))?+|👨(([🏻-🏿](\\x{200D}(([⚕-⚖✈]\\x{FE0F}|[🌾🍳🎓🎤🎨🏫🏭💻-💼🔧🔬🚀🚒🦰-🦳])))?+|\\x{200D}(([⚕-⚖✈]\\x{FE0F}|👦(\\x{200D}👦)?+|👧(\\x{200D}[👦-👧])?+|[👨-👩]\\x{200D}((👦(\\x{200D}👦)?+|👧(\\x{200D}[👦-👧])?+))|❤\\x{FE0F}\\x{200D}((💋\\x{200D}👨|👨))|[🌾🍳🎓🎤🎨🏫🏭💻-💼🔧🔬🚀🚒🦰-🦳]))))?+|👩(([🏻-🏿](\\x{200D}(([⚕-⚖✈]\\x{FE0F}|[🌾🍳🎓🎤🎨🏫🏭💻-💼🔧🔬🚀🚒🦰-🦳])))?+|\\x{200D}(([⚕-⚖✈]\\x{FE0F}|👦(\\x{200D}👦)?+|👧(\\x{200D}[👦-👧])?+|👩\\x{200D}((👦(\\x{200D}👦)?+|👧(\\x{200D}[👦-👧])?+))|❤\\x{FE0F}\\x{200D}((💋\\x{200D}[👨-👩]|[👨-👩]))|[🌾🍳🎓🎤🎨🏫🏭💻-💼🔧🔬🚀🚒🦰-🦳]))))?+|[⌚-⌛⏩-⏬⏰⏳◽-◾☔-☕♈-♓♿⚓⚡⚪-⚫⚽-⚾⛄-⛅⛎⛔⛪⛲-⛳⛵⛺⛽✅✨❌❎❓-❕❗➕-➗➰➿⬛-⬜⭐⭕🀄🃏🆎🆑-🆚🈁🈚🈯🈲-🈶🈸-🈺🉐-🉑🌀-🌠🌭-🌵🌷-🍼🍾-🎄🎆-🎓🎠-🏁🏅-🏆🏈-🏉🏏-🏓🏠-🏰🏸-🐾👀👄-👅👑-👥👪-👭👹-👻👽-💀💄💈-💩💫-📼📿-🔽🕋-🕎🕐-🕧🖤🗻-🙄🙈-🙊🚀-🚢🚤-🚳🚷-🚿🛁-🛅🛐-🛒🛫-🛬🛴-🛹🤐-🤗🤝🤠-🤥🤧-🤯🤺🥀-🥅🥇-🥰🥳-🥶🥺🥼-🦢🦰-🦴🦷🧀-🧂🧐🧠-🧿])

Dooming us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane:

Different programs may not identify emoji as being the same, which can clearly cause problems. For example, hashtags are commonly treated as identical even if they differ by case (#foo = #FOO = #Foo = #fOO). But hashtags #🎅🏿 and #🎅🏻 and #🎅 may be treated the same on some systems (≅ #santa, but language-neutral), but treated differently on others.

Previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously, previously.

Leave Britney's Command and Control Server Alone!

jwz — Wed, 07 Jun 2017 00:23:12 +0000

Turla's watering hole campaign: An updated Firefox extension abusing Instagram

The extension uses a bit.ly URL to reach its C&C, but the URL path is nowhere to be found in the extension code. In fact, it will obtain this path by using comments posted on a specific Instagram post. The one that was used in the analyzed sample was a comment about a photo posted to the Britney Spears official Instagram account.
The extension will look at each photo's comment and will compute a custom hash value. If the hash matches 183, it will then run this regular expression on the comment in order to obtain the path of the bit.ly URL:
(?:\\u200d(?:#|@)(\\w)
Looking at the photo's comments, there was only one for which the hash matches 183. This comment was posted on February 6, while the original photo was posted in early January. Taking the comment and running it through the regex, you get the following bit.ly URL:
http://bit.ly/2kdhuHX
Looking a bit more closely at the regular expression, we see it is looking for either @|# or the Unicode character \200d. This character is actually a non-printable character called 'Zero Width Joiner', normally used to separate emojis. Pasting the actual comment or looking at its source, you can see that this character precedes each character that makes the path of the bit.ly URL:
smith2155<200d>#2hot ma<200d>ke lovei<200d>d to <200d>her, <200d>uupss <200d>#Hot <200d>#X

Previously, previously, previously.

Zalgo Rly

jwz — Thu, 21 Apr 2016 00:24:33 +0000

Previously, previously, previously.

Piet Mondrian is Turing Complete

jwz — Fri, 08 Jan 2016 11:36:52 +0000

Program code will be in the form of abstract art. Piet uses 20 distinct colours. The 18 colours in the first 3 rows of the table are related cyclically.

Lightness change

Hue change None 1 Darker 2 Darker

None push pop

1 Step add subtract multiply

2 Steps divide mod not

3 Steps greater pointer switch

4 Steps duplicate roll in(number)

5 Steps in(char) out(number) out(char)

The Piet language interpreter begins executing a program in the colour block which includes the upper left codel of the program. The interpreter maintains a Direction Pointer (DP), initially pointing to the right. The DP may point either right, left, down or up. The interpreter also maintains a Codel Chooser (CC), initially pointing left. The CC may point either left or right. The directions of the DP and CC will often change during program execution. As it executes the program, the interpreter traverses the colour blocks of the program under the following rules: [...]

	Lightness change
Hue change	None	1 Darker	2 Darker
None		push	pop
1 Step	add	subtract	multiply
2 Steps	divide	mod	not
3 Steps	greater	pointer	switch
4 Steps	duplicate	roll	in(number)
5 Steps	in(char)	out(number)	out(char)

Hello World

Print "Piet"

Primes

Brainfuck interpreter

Conway's Life

The Esoteric programming language page is remarkably complete -- I didn't know about a third of these!

I can't even decide if this counts as steganography. Imagine the computational load that could be lurking, un-executed, in museums around the world!

Previously, previously, previously, previously, previously, previously, previously, previously, previously.

Linty linty

jwz — Sun, 24 May 2015 06:47:15 +0000

I have spent the last few days down in the PHP Mines. Those fumes are really toxic. I'm pretty sure I've contracted pleurisy from inhaling Cheetos dust.

So DNA Lounge has an online store, which is an enormous, and ancient, and shitty, pile of PHP code. I have often characterized it as "it's a pile of shit, but it's our pile of shit."

Well, now that I'm opening a new nightclub, I need to make that code be able to function when installed on a domain whose name is not "dnalounge.com". As you can expect, this has been a process of discovering all kinds of hardcoded assumptions lurking in the bowels. It's like moving to a new house after you've been dug in for a decade: look at all this crap stuffed in the back of the closet that you didn't remember was there!

So there has been a lot of global search and replace, and a lot of variables being added and moved around, and that means destabilization, and that's bad, mmkay?

So I wrote a Perl program to do static analysis of PHP. And it parses PHP using regular expressions because fuck you that's why.

For your entertainment -- because I sincerely doubt that anyone who is not me will ever use this without lulz in their heart -- I give you jwz-php-lint.pl.

Now you have two problems, and one of them is Flash.

jwz — Mon, 16 Feb 2015 02:18:29 +0000

Project Zero: (^Exploiting)\s(CVE-2015-0318)\s(in)\s*(Flash$)

Quick summary - it's a bug in the PCRE regex engine as used in Flash. Spoiler: it's exploitable. [...]
Below is what happens when we compile a regex that combines the \c escape sequence (which is intended to match a single ASCII character) with a multibyte UTF-8 character. A simple trigger for the bug is '\\c\xd0\x80+', below. [...]
So clearly something has gone wrong... The question is now how to leverage this invalid bytecode to get code execution.

Previously, previously, previously, previously, previously, previously, previously, previously.

"jwz has more than two problems."

jwz — Tue, 28 Oct 2014 19:17:12 +0000

Why use PHP/RoR/Django for your CMS when you can use Perl?

(In fact, I have an abundance of problems. But Django ain't one.)

Previously, previously, previously.

We live in a magical future where "strings" is exploitable.

jwz — Sun, 26 Oct 2014 00:26:42 +0000

What is this I can't even.

Many shell users, and certainly most of the people working in computer forensics or other fields of information security, have a habit of running /usr/bin/strings on binary files originating from the Internet. Their understanding is that the tool simply scans the file for runs of printable characters and dumps them to stdout - something that is very unlikely to put you at any risk.
It is much less known that the Linux version of strings is an integral part of GNU binutils, a suite of tools that specializes in the manipulation of several dozen executable formats using a bundled library called libbfd. Other well-known utilities in that suite include objdump and readelf. [...]
In any case: the bottom line is that if you are used to running strings on random files, or depend on any libbfd-based tools for forensic purposes, you should probably change your habits.

Previously, previously, previously, previously, previously, previously.

Dooming us to inhuman toil, etc, etc.

jwz — Sun, 05 Oct 2014 07:07:52 +0000

You'll be happy to know that this weekend I've dragged XScreenSaver kicking and screaming into 2003 and have made all the hacks that load text able to properly display UTF-8 input, and it's even anti-aliased.

But don't worry: while doing so, it still parses HTML using regular expressions. I would never re-make a classic hit like that.

Previously, previously, previously.

So this happened.

jwz — Mon, 19 May 2014 03:15:27 +0000

What is meant by "Now you have two problems"?
There is a popular quote by Jamie Zawinski:
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
HOW IS BABBY FORMED?

I mention this only because it gives me a chance to tell you about one of the saddest features of my Twitter feed, which is that every couple of weeks, someone out there in the dunderweb @-mentions me with this super clever new variant they've just now thought up all by themselves on either "now you have two problems" or the Law of Software Envelopment (the latter usually sounds something like "Hurf durf cloud durf Facebook hurf durf expands hurf durf git".)

Stop.

Stop. Please stop.

Staaaaahhhhhhhhhppp.

You are making a "GOT MILK"* joke here, people. It's the Twenty First God Damned Century. It is no longer the Nineteen Hundred and Nineties. It's time for some new jokes!

(Though I did get a kick out of this one for a minute.)

Previously, previously, previously.

Accidentally Turing-Complete

jwz — Tue, 25 Mar 2014 17:50:57 +0000

Some things were not supposed to be Turing-complete. This is a collection of such accidents.

Pokemon Yellow: The interesting point about this speedrun is the bug it exploits. Turns out the game logic itself is Turing-complete in the sense that you can rewrite the assembly itself with game actions. For example, someone turned the game into a MIDI player.
MediaWiki Templates: In MediaWiki you can define templates. Since they provide recursion, you can apparently implement lambda calculus.

Previously, previously.

Canonical Comment Spam

jwz — Fri, 26 Apr 2013 02:54:29 +0000

"Spammer just exposed his template and the whole thing showed up in my comments."

{ {I have|I've} been {surfing|browsing} online more than {three|3|2|4} hours today, yet I never found any interesting article like yours. {It's|It is} pretty worth enough for me. {In my opinion|Personally|In my view}, if all {webmasters|site owners|website owners|web owners} and bloggers made good content as you did, the {internet|net|web} will be {much more|a lot more} useful than ever before.| I {couldn't|could not} {resist|refrain from} commenting. {Very well|Perfectly|Well|Exceptionally well} written!| {I will|I'll} {right away|immediately} {take hold of|grab|clutch|grasp|seize|snatch} your {rss|rss feed} as I {can not|can't} {in finding|find|to find} your {email|e-mail} subscription {link|hyperlink} or {newsletter|e-newsletter} service. Do {you have|you've} any? {Please|Kindly} {allow|permit|let} me {realize|recognize|understand|recognise|know} {so that|in order that} I {may just|may|could} subscribe. Thanks.|
[ ...and so on, for hundreds more lines. ]

The Turing Police say you have more than two problems.

jwz — Thu, 29 Dec 2011 02:37:13 +0000

Occupy Babel!

Hard-to-parse protocols require complex parsers. Complex, buggy parsers become weird machines for exploits to run on. Help stop weird machines today: Make your protocol context-free or regular!
Protocols and file formats that are Turing-complete input languages are the worst offenders, because for them, recognizing valid or expected inputs is UNDECIDABLE: no amount of programming or testing will get it right.
A Turing-complete input language destroys security for generations of users. Avoid Turing-complete input languages!
Ensure computational equivalence of protocol endpoints: use only regular and context-free protocols!

Needless to say, you also doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane.

You doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane.

jwz — Sat, 26 Nov 2011 00:11:24 +0000

RegEx match open tags except XHTML self-contained tags:

You can't parse [X]HTML with regex. Because HTML can't be parsed by regex. Regex is not a tool that can be used to correctly parse HTML. As I have answered in HTML-and-regex questions here so many times before, the use of regex will not allow you to consume HTML. Regular expressions are a tool that is insufficiently sophisticated to understand the constructs employed by HTML. HTML is not a regular language and hence cannot be parsed by regular expressions. Regex queries are not equipped to break down HTML into its meaningful parts. so many times but it is not getting to me. Even enhanced irregular regular expressions as used by Perl are not up to the task of parsing HTML. You will never make me crack. HTML is a language of sufficient complexity that it cannot be parsed by regular expressions. Even Jon Skeet cannot parse HTML using regular expressions. Every time you attempt to parse HTML with regular expressions, the unholy child weeps the blood of virgins, and Russian hackers pwn your webapp. Parsing HTML with regex summons tainted souls into the realm of the living. HTML and regex go together like love, marriage, and ritual infanticide. The
cannot hold it is too late. The force of regex and HTML together in the same conceptual space will destroy your mind like so much watery putty. If you parse HTML with regex you are giving in to Them and their blasphemous ways which doom us all to inhuman toil for the One whose Name cannot be expressed in the Basic Multilingual Plane, he comes. HTML-plus-regexp will liquify the nerves of the sentient whilst you observe, your psyche withering in the onslaught of horror. Rege̿̔̉x-based HTML parsers are the cancer that is killing StackOverflow it is too late it is too late we cannot be saved the trangession of a chi͡ld ensures regex will consume all living tissue (except for HTML which it cannot, as previously prophesied) dear lord help us how can anyone survive this scourge using regex to parse HTML has doomed humanity to an eternity of dread torture and security holes using regex as a tool to process HTML establishes a breach between this world and the dread realm of c͒ͪo͛ͫrrupt entities (like SGML entities, but more corrupt) a mere glimpse of the world of regex parsers for HTML will instantly transport a programmer's consciousness into a world of ceaseless screaming, he comes~~, the pestilent sl~~ithy regex-infection will devour your HTML parser, application and existence for all time like Visual Basic only worse he comes he comes do not fight he com̡e̶s, ̕h̵is un̨ho͞ly radiańcé destro҉ying all enli̍̈́̂̈́ghtenment, HTML tags lea͠ki̧n͘g fr̶ǫm ̡yo͟ur eye͢s̸ ̛l̕ik͏e liquid pain, the song of re̸gular expre~~ssion parsing~~ will extinguish the voices of mortal man from the sphere I can see it can you see ̲͚̖͔̙î̩́t̲͎̩̱͔́̋̀ it is beautiful the final snuffing of the lies of Man ALL IS LOŚ͖̩͇̗̪̏̈́T ALL IS LOST the pon̷y he comes he c̶̮om~~es he co~~mes the ichor permeates all MY FACE MY FACE ᵒh god no NO NOO̼OO NΘ stop the an*̶͑̾̾̅ͫ͏̙̤g͇̫͛͆̾ͫ̑͆l͖͉̗̩̳̟̍ͫͥͨe̠̅s ͎a̧͈͖r̽̾̈́͒͑e not rè̑ͧ̌aͨl̘̝̙̃ͤ͂̾̆ ZA̡͊͠͝LGΌ ISͮ̂҉̯͈͕̹̘̱ TO͇̹̺ͅƝ̴ȳ̳ TH̘Ë͖́̉ ͠P̯͍̭O̚N̐Y̡ H̸̡̪̯ͨ͊̽̅̾̎Ȩ̬̩̾͛ͪ̈́̀́͘ ̶̧̨̱̹̭̯ͧ̾ͬC̷̙̲̝͖ͭ̏ͥͮ͟Oͮ͏̮̪̝͍M̲̖͊̒ͪͩͬ̚̚͜Ȇ̴̟̟͙̞ͩ͌͝S̨̥̫͎̭ͯ̿̔̀ͅ

Mork keeps on giving: When the database worms eat into your murder trial

jwz — Tue, 19 Jul 2011 20:27:25 +0000

You may recall that back in 2004, I tried to extract the recently-visited URLs from Mozilla's history database, re-discovered some major, major insanity, and did some colorful swearing. See "When the database worms eat into your brain", the halfassed perl script I eventually came up with, RESOLVED WONTFIX bug 241438, and where it all started, "Netscape Mail Summary Files", 1998.

(For some reason, one of the wikitards thought that this bug report was of such significance to the Global Repository of All Human Knowledge that there's an entire sub-section on my Wikipedia page about it. I spent perhaps a grand total of 24 hours of my life thinking about this -- ok maybe 25 by now -- and this is what gets memorialized? WTF. Seriously. W the F.)

Anyway, you may be interested to know that seven years after that bug report, and thirteen years after it should have happened:

@chrisblizzard 5 Jul 2011
I'm told that the last of mork has been excised from the Mozilla tree. (this is kind of for @shaver, but really for @jwz)

But what brings us here today is a gentle reminder that when you write code this bad, you can actually kill people.

I'm leaving the DB-dump images in the following quote as a reminder of just how insane this code was. Think of these as skulls on sticks at the edge of the wasteland, saying "Never pass this way again".

Digital Evidence Discrepancies -- Casey Anthony Trial

The digital forensic evidence in this case is of particular interest to me as it involved the recovery and analysis of a Mozilla Firefox history database. The Internet history records within this database turned out to be extremely important to the prosecution case as the existence of Google searches relating to "chloroform" and other possibly relevant records prior to the child's disappearance could have indicated premeditation. This, of course, could have meant the difference between a conviction for murder in the first degree and manslaughter if found guilty. The State of Florida also has the death penalty as a punishment option for capital crimes.
During a keyword search of Anthony's computer, a hit was found for the word "chloroform". The hit was identified in what appeared to be a Mork database belonging to Mozilla Firefox. The file was identified as residing in unallocated clusters, and rather surprisingly, is reported to have been intact. Furthermore, all of the blocks belonging to the file were said to be contiguous. [...]
He pointed out the discrepancy between the first analysis the sheriff’s office did that showed one visit to a website about chloroform and an analysis done later with a second program that appeared to show 84 visits. However, according to Baez, the first report showed a progression that made it clear that the 84 visits were actually to MySpace. This was a major discrepancy with critical digital evidence presented in an extremely serious trial. [...]
The Mork record containing "http://www.sci-spot.com/Chemistry/chloroform.htm" is identified as record 174EF. The Index record from the original file is highlighted and shown in Figure 10 below.

Figure 10
The entire record is contained within square brackets. The highlighted line above shows the full record. The first field 82 ("URL") is stored in cell 27F4B, as shown in Figure 11.

Figure 11
The second field 84 ("LastVisitDate") is stored in cell 27F4C, as shown in Figure 12 (2008-03-21 19:16:34 UTC / 2008-03-21 15:16:34 Local Time). Once again, this integer represents the number of micro-seconds from the 1st January 1970, 00:00:00 UTC.

Figure 12
The third field 85 ("FirstVisitDate") is stored in cell 27F4C. This is the same cell value as for ("LastVisitDate") and indicates this is the first visit to this web site during the scope of the current recorded history. The First and Last visit times are the same.
The fourth field 83 (“Referrer”) is stored in cell 27F49, as shown in Figure 13.

Figure 13
There are two critical points to make with this record. Firstly, there is no field 86 ("VisitCount") therefore this URL has only been visited once (not 84 times). This is further corroborated by the fact that field 85 ("FirstVisitDate") shows the exact same date/time as the "LastVisitDate". The second point is that the visit was recorded at 15:16:34 hours (local time) and NOT at 15:16:13 hours as was stated during the trial (from the report produced by the second forensic tool).

(Let me emphasize that those images above are not hex dumps or something: that's the actual, literal text of this file format!)

Previously, previously.

State of the Onion, and screen savers

jwz — Fri, 20 Aug 2004 22:28:00 +0000

The State of the Onion #8: Larry Wall's "State of the Onion" speeches are always really entertaining: they're nominally about the current state of Perl, but really they're about all kinds of things. The latest one was mostly about his recent health problems, but also about community, cognition, and design. This time, instead of slides, he ran screen savers and related the behavior of each to what he was talking about.

[BSOD]
This particular screensaver fools me more often than I care to admit. The problem is that the more computers you've used, the more different kinds of crashes you've seen. And mentally, you classify them all in the "Oh, shit!" category, which is a category the brain is very efficient at processing.
On the other hand, the part of your brain that says "Hey, that's the crash screen for a different operating system, dufus!" -- that part operates at a much slower pace. The brain is chock full of shortcuts, and orthogonality be screwed. Optimizers cheat, and sometimes they get caught cheating. With this screensaver, you can catch your own brain's optimizer cheating.

perl and unicode go together like apples and razor blades

jwz — Tue, 16 Sep 2003 20:06:00 +0000

That scrmable thing has really been making the rounds: I've seen the text translated into three or four other (human) languages now, not to mention all the people writing their own scripts in their marginalized geek-language du jour.

But my script was malfunctioning for a bunch of people, and I finally figured out why. Fucking Unicode again. If $LANG contains "utf8" (which is the default on recent Red Hat systems), then "^\w" doesn't work right, among other things. Check this out:

setenv LANG en_US echo -n "foo.bar" | \ perl -e '$_ = <>; print join (" | ", split (/([^\w]+)/)) . "\n";'

===> "foo | . | bar" (right)

setenv LANG en_US.utf8 echo -n "foo.bar" | \ perl -e '$_ = <>; print join (" | ", split (/([^\w]+)/)) . "\n";'

===> "foo.bar" (wrong!)

It works fine in both cases if you do $_ = "foo.bar" instead of reading it from stdin.

perl-5.8.0-88, Red Hat 9. Hate.