Regular Expressions

Beyond the fundamentals

Peter Makholm

Copenhagen.pm

Disclaimer

⁰⁾This is to be interpreted as described in RFC 2119

Beyond the fundementals

So you learned the basics, what's next?

• Practise
• Advanced features
• Practise
• Keep it Simple, Stupid
• Help from CPAN
• Practise

Advanced features

• Look around assertions
• Captures and backreferences
• Global matches
• Independent subexpressions
• The qr// operator

General syntax

• New parentheses on the form: (?X...)
• In general not capturing
• Non capturing parentheses: (?:...)

Look around assertions

• Zero width assertions, like $ and \b
• Look behind or look ahead. Positive and negative assertions
  • positive lookahead: (?=...)
  • negative lookahead: (?!...)
  • positive lookbehind: (?<=...)
  • negative lookbehind: (?<!...)
Look behinds have to be fixed width

- examples

• Matching a word boundary:
  (?<=\w)(?=\W|$)|(?<=^|\W)(?=\w)
• Parsing right justified tables
  (\d*)(?<=.{17})
• Replacing 'Alice' with 'Anita' except in the famous 'Alice and Bob'
  s/Alice(?! and Bob)/Anita/g
• Lines talking about Perl sucking?
  Perl(?=.*sucks)

Captures and backreferences

Perl 5.10 provides some new options for captures and back references:

• Named capture:
  /(?<hour>\d\d):(?<minute>\d\d)/
  • captures to %+ (ie. $+{hour} etc.)
  • backreferences as \g{hour}
• Relative backreferences: \g{-1}
• Alternative numbering:
  /(?|(\d\d)(\d\d)|(\d\d):(\d\d))/
  • hour and minute is in $1 and $2 no matter what.

Global matches

With global matches enabled, Perl keeps track of the position it has reached while performing matches on a string

• //g - enable global matching
• //c - don't reset pos() on failure
• /\G.../ - anchor at pos()
• pos() = 12 - setting pos()

- A simple lexer

while(<>) {
    print(" lowercase") and redo if /\G[a-z]+\s+/gc;
    print(" uppercase") and redo if /\G[A-Z]+\s+/gc;
    print(" digits")    and redo if /\G[0-9]+\s+/gc;
    print(" mixed")     and redo if /\G[a-zA-Z0-9]+\s+/gc;
    print(" noise")     and redo if /\G[^a-zA-Z0-9]+\s+/gc;
    print "\n";
}

Independent subexpressions

• (?>...)
• "grab all you can, and do not give anything back"
• Faster failures:
  • Matching 'Subject' against ^\w+:
  • Prevent backtracking: ^(?>\w+):
• Prevent matches
  • Match 'aaab' against ^(?>a*)ab

Possessive quantifiers

• a*+ is equivalent to (?>a*)
• a++ is equivalent to (?>a+)
• a?+ is equivalent to (?>a?)
• a{min,max}+ is equivalent to (?>a{min,max})

The qr// operator

The qr// operator turns the regexp into a value you can store and parse around

@regexp = ( qr/foo/, qr/bar/, qr/baz/ );
while(defined( $line = <> )) {
    print $line if grep { $line =~ $_ } for @regexp;
}

Keep It Simple, Stupid

KISS, the alternative

However, don't end up like this

qr/
   (((0[48]|[2468][048]|[13579][26])00|\d\d(0[48]|[2468]
   [048]| [13579] [26])) |(( [02468] [1235679]|  [13579]
   [01345789])00|\d\d([02468][1235679]|[13579][01345789]
   ))(?!(-|)02\g{-1}29))(?<sep>-|)(02(?!\g{sep}3)|0[469]
   (?!\g{sep}31)|11(?!\g{sep} 31)|0[13578]|1[02])\g{sep}
   (0[1-9]|[12][0-9]|3[01])
  /x;

Add spaces and comments

For improved readability, use /x to add whitespaces and comments:

$isLeapYear = 
  qr/( 
    # Either century divisible by 400:
      ( 0[48] | [2468][048] | [13579][26] ) 00
    # Or year divisible by 4, but not a century 
      \d\d ( 0[48] | [2468][048] | [13579][26] )
     )/x;

The missing and operator

Often people want to match "this" and "that" in a single regexp. It's possible, but ...

• print if /^(?=.*foo)(?=.*bar)/;
• print if /foo/ && /bar/;

Which one is easier to read?

Extract and validate in Perl

Remember this?

qr/
   (((0[48]|[2468][048]|[13579][26])00|\d\d(0[48]|[2468]
   [048]| [13579] [26])) |(( [02468] [1235679]|  [13579]
   [01345789])00|\d\d([02468][1235679]|[13579][01345789]
   ))(?!(-|)02\g{-1}29))(?<sep>-|)(02(?!\g{sep}3)|0[469]
   (?!\g{sep}31)|11(?!\g{sep} 31)|0[13578]|1[02])\g{sep}
   (0[1-9]|[12][0-9]|3[01])
  /x;

It validates dates on the form 'YYYY-MM-DD' or 'YYYYMMDD'.
Excercise: Add support for 'dd/mm/yyyy'

- Validating dates

Try this instead:

my @daysInMonth = (  0, 31, 28, 31, 30, 31, 
                    30, 31, 31, 30, 31, 30, 31);

if( /(?<y>\d\d\d\d)-(?<m>\d\d)-(?<d>\d\d)/ ) {
  return if $+{m} == 0 
         || $+{d} == 0;
         || $+{m} > 12;
  return 1 if $+{d} <= $daysInMonth[ $+{m} ];
  return 1 if $+{m} = 2 
           && $+{d} = 29
           && leapYear( $+{y} );
  return;
}

Never parse HTML or XML

• Too many pesky corner cases
• Too many places that shouldn't be parsed (remember comments or <![CDATA[...]]> ?)
• Too many usable alternatives
  • HTML::Parser
  • HTML::TreeBuilder
  • XML::LibXML

Regexp::Common: Don't reinvent

Ready to use regexpes for a lot of common cases:

• URIs
• quoted strings
• numbers
• time and dates

Test::Regexp: Testing Regular expressions

use Test::Regexp 'no_plan';

match    subject      => "Foo bar",
         keep_pattern => qr /(?<first_word>\w+)\s+(\w+)/,
         captures     => [[first_word => 'Foo'], ['bar']];

no_match subject      => "Baz",
         pattern      => qr /Quux/;

Questions?