Parsing "real" C code in ruby using CAST.

CAST - parser for C written in Ruby is a pretty nice piece of machinery. One drawback it has - it does not have the preprocessor. And it chokes on "#" origin file marks left by CPP. So, as a first step to making it more fun for real-world use, I made it more user-friendly when it comes to parsing the CPP-digested files.

The algorithm (if I may call it like that) is dead simple: read the lines from CPP-digested file,
and create a "compressed" content, with empty lines and #-lines removed. Also need to do some gsub()-style replacement of some gcc-isms before stuffing them into the compressed content. For each line of compressed content, record the original line#, as well as track the pre-CPP filename and line#. When/if CAST barfs, catch the exception and parse the line# from the message. Then lookup in the info that you've generated previously - and generate your own exception, this time supplying the miscellaneous goodies like the pre-processed file and its line number.

With the resulting wrapper parser it seems like I can parse a lot of my personally written sources. The cool part is that it catches where I have a bit "stretched" the gcc's kindness - cast's parser is stricter than GCC. I'm not sure if it is "pure C99", but it's a good thing to have something like this.

The only tiny gotcha I got caught with while coding this little 80-line piece: line number == array index + 1. I named one variable "linenumber" whereas really it was counting the array index - so indeed shot myself in the foot a bit later. Sometimes I wonder why don't we all start counting from zero, really.

Update: here's the ruby hack that allows the CAST to not barf on the preprocessed file. Again - the "real_code.c" is the file *after* the cpp processing. One could write their own preprocessor indeed, but I just reused cpp.

require 'rubygems'
require 'cast'


#
# "prepare" a line - strip any of the gcc internal stuff from it,
# and any other things we do not understand.
#

def prepare_line(str)
    line = str.strip
    line = line.gsub(/\(\(__.*\)\)/, "").gsub(/__(attribute|extension)__/, "")
    line = line.gsub(/__const /, "const ")
    line = line.gsub(/__restrict /, "")
    line = line.gsub(/__asm__\s*\(.*\)/, "")
    line
end

#
# "prepare" the preprocessed output in array "lines"
# taken from file fname (filename used only for error messages)
#
def prepare_lines(fname, lines)
  # blob that will be parseable by CAST
  real_text = Array.new
  # info about the lines in the blob
  real_info = Array.new
  original_file = fname
  original_lineno = 1
  lines.each_with_index do |line, idx|
    real_line = prepare_line(line)

    if real_line =~ /^#\s+(\d+)\s+"([^"]+)"((?:\s+.*)?)$/
      num = $1.to_i
      fn = $2
      misc = $3
      original_file = fn
      original_lineno = num # minus something ?
      # puts "== set #{original_file}:#{original_lineno}"
      next
    else
      if real_line =~ /^#/
        raise real_line
      end
    end
    unless real_line =~ /^$/
      real_text << real_line
      real_info << { :file => original_file, :lineno => original_lineno,
                   :debug_file => fname, :debug_lineno => idx+1 }
      # puts "#{original_file}:#{original_lineno}:#{real_line}"
    end

    original_lineno += 1
  end
  [ real_text, real_info ]
end
def parse_c(fname)
  lines = File.open(fname).read.split("\n")

  text, info = prepare_lines(fname, lines)
  blobtext = text.join("\n")

  parser = C::Parser.new
  parser.type_names << "__builtin_va_list"
  parser.type_names << "double"
  tree = nil

  begin
    print "Parse start...\n"
    tree = parser.parse(blobtext)
    print "Parse end...\n"
  rescue Exception => e
    puts "Got exception: #{e.inspect}"
    if e.message =~ /^(\d+):(.*)$/
      errline = $1.to_i
      errmsg = $2
      ei = info[errline-1]
      src = text[errline-2 .. errline+1].join("\n")
      # puts "source lines: #{src}"

      raise "Error in #{ei[:file]}:#{ei[:lineno]} (#{ei[:debug_file]}:#{ei[:debug_lineno]}) : #{errmsg}"

    else
      raise "unparseable error message from parser"
    end
  end


  tree
end

tree = parse_c("real_code.c")

# puts tree.to_s
p tree