License Public Domain
Lines 17
Keywords
lexer utils (1)
Permissions
Owner: Stou S.
Group Owner: iCapsid
Viewable by Everyone
Editable by All Siafoo Users
Hide
Easily highlight source code for your blog with our Syntax Highlighter. Join Siafoo Now or Learn More

Lexer builtins creator Atom Feed 0

In Brief Given a file with a list of keywords this snippet will create a regex block. I used it for creating the the 'builtins' regex for various lexers.
# 's
 1from sets import Set
2
3data = open('in_file.txt').read()
4
5# Make the words unique
6kwords_list = list(Set(data.split('|')))
7kwords_list.sort()
8
9out_file = open('out_file.txt', 'w')
10line_limit = 100
11char_count = 0
12temp_list = []
13
14for k in kwords_list:
15 if len(k) + char_count > line_limit:
16 out_file.write("r'%s|'\n" % '|'.join(temp_list))
17 temp_list = []
18 char_count = 0
19
20 char_count += len(k)
21 temp_list.append(k)
22
23out_file.write("r'%s|'\n" % '|'.join(temp_list))

Given a file with a list of keywords this snippet will create a regex block. I used it for creating the the 'builtins' regex for various lexers.