License Public Domain
Lines 17
Keywords
lexer utils (1)
Permissions
Owner: Stou S.
Group Owner: iCapsid
Viewable by Everyone
Editable by All Siafoo Users
Hide
Stay up to dateembedded code automagically updates, each snippet and article has a feed Join Siafoo Now or Learn More

Lexer builtins creator Atom Feed 0

In Brief Given a file with a list of keywords this snippet will create a regex block. I used it for creating the the 'builtins' regex for various lexers.
# 's
 1from sets import Set
2
3data = open('in_file.txt').read()
4
5# Make the words unique
6kwords_list = list(Set(data.split('|')))
7kwords_list.sort()
8
9out_file = open('out_file.txt', 'w')
10line_limit = 100
11char_count = 0
12temp_list = []
13
14for k in kwords_list:
15 if len(k) + char_count > line_limit:
16 out_file.write("r'%s|'\n" % '|'.join(temp_list))
17 temp_list = []
18 char_count = 0
19
20 char_count += len(k)
21 temp_list.append(k)
22
23out_file.write("r'%s|'\n" % '|'.join(temp_list))

Given a file with a list of keywords this snippet will create a regex block. I used it for creating the the 'builtins' regex for various lexers.