ic blogs

Notes on ply

十二月 25th, 2011 | ic

introduction

ply 是python版的lex 和yacc ，尝试了下document里面的calculator例子，感觉很好用，非常的python，呵呵。[1]有作者写的详细的文档。pycparser 是在ply上实现的，可以parse c和生成ast（abstract syntax tree) ，是个非常好的参考。本文的大部分内容源自[1]

ply.lex

首先要定义变量tokens, 这个变量是必须的有的。

有两种匹配token的方法，一种是简单的，如t_PLUS = r’\+’，另外一种token在遇到的时候，要做写些相应的处理，比如对于nummber来说，需要得到的它值，这个需要定义一个函数来处理。

ply.lex 使用函数的文档, 即 fun.__doc__作为匹配的正则表达式，还有另外一种方法是采用@Token修饰符。(详见[1])。这里的 r’\d+’ 是token NUMBER所对应的正则表达式. 这里面lex规则的前缀t_也是必须的。

另外ply.lex.input（text)可以输入文本，然后从ply.lex.token()里面取token.

  tokens = (
        "NUMBER",
        "PLUS",
        "MINUS",
        "DIVIDE",
        "TIMES",
        "LPAREN",
        "RPAREN",
    )
    #regular expression rule for simple tokens
    t_PLUS = r'\+'

    #a regular express rule with some action code. fun.__doc__ is related regular expression
    def t_NUMBER(self, t):
        r'\d+'
        t.value = int(t.value)
        return t

Demo

这里面Calclexer.build没有放到__init__的原因，见[1]:

you should construct the lexer from an instance of the class, not the class object itself. This is because PLY only works properly if the lexer actions are defined by bound-methods.

另外yacc里面的规则是可以合并的，这里没合并。[1]有介绍。

import ply.lex as lex
import ply.yacc as yacc
class Calclexer:
    #list of token names.
    tokens = (
        "NUMBER",
        "PLUS",
        "MINUS",
        "DIVIDE",
        "TIMES",
        "LPAREN",
        "RPAREN",
    )

    #regular expression rule for simple tokens
    t_PLUS = r'\+'
    t_MINUS = r'-'
    t_TIMES = r'\*'
    t_DIVIDE = r'/'
    t_LPAREN = r'\('
    t_RPAREN = r'\)'

    #a regular express rule with some action code. fun.__doc__ is related regular expression
    def t_NUMBER(self, t):
        r'\d+'
        t.value = int(t.value)
        return t

    def t_newline(self, t):
        r'\n+'
        t.lexer.lineno += len(t.value)
    #a string containsing ignored characters (space and tabs)
    t_ignore = ' \t'

    #error handling rules
    def t_error(self, t):
        print "illegal charcter '%s'" % t.value[0]
        t.lexer.skip(1)
        return

    #Build the lexer
    def build(self, **kwargs):
        self.lexer = lex.lex(module=self, **kwargs)

    def input(self, data):
        self.lexer.input(data)
        return

    def token(self):
        return self.lexer.token()

    def test(self, data):
        self.lexer.input(data)
        while True:
            token = self.lexer.token()
            if not token: break
            print token

class CalcParser:
    def __init__(self,  **kwargs):
        self.calclex = Calclexer()
        self.calclex.build()

        self.tokens = self.calclex.tokens
        self.calcparser = yacc.yacc(module=self)

    def p_expression_plus(self, p):
        'expression : expression PLUS term'
        p[0] = p[1] + p[3]

    def p_expression_minus(self, p):
        r"expression : expression MINUS term"
        p[0] = p[1] - p[3]

    def p_expression_term(self, p):
        "expression : term"
        p[0] = p[1]

    def p_term_div(self, p):
        "term : term DIVIDE factor"
        p[0] = p[1] / p[3]

    def p_term_times(self, p):
        "term : term TIMES factor"
        p[0] = p[1] * p[3]

    def p_term_factor(self, p):
        'term : factor'
        p[0] = p[1]

    def p_factor_num(self, p):
        "factor : NUMBER"
        p[0] = p[1]

    def p_factor_expr(self, p):
        "factor : LPAREN expression RPAREN"
        p[0] = p[2]

    def p_errorr(self, p):
        print "syntax error in input"

    def parse(self, text):
        return self.calcparser.parse(text, lexer=self.calclex)

def main():
    parser =CalcParser()
    while True:
        try:
            s = raw_input('calc >')
        except EOFError:
            break
        if not s: continue
        result = parser.parse(s)
        print result

if __name__ == "__main__":
    main()

References

1. 主要的文档
2. pycparser 主页
Posted in python, 技术 | No Comments »