Meet people who work on similar things as you – get help if you need it Join Siafoo Now or Learn More

The Basics

over 11 years ago

Before I started this blog, I had already done a decent chunk of work on the project - everything up to and including a few basic GUI elements and the functions that make the program able to dynamically find, read and timestamp (or just 'stamp' as I call it) a logfile as it is being written. The program has gone through a handful of rewrites already as I discover new things about Python that would make it more efficient.

Finding A File

With this analyzer, there are two ways to find a file to read from. The user can specify a single file, or the user can instead simply point to a directory and the program will find and use the latest modified combat log in that directory. Both of these tasks are trivial, but I had some concern over how the second method worked because I implemented it in a loop that would check the folder every second or so and then switch if a brand new logfile was created in the middle of reading the old one. I did this to enable the user to switch logfiles without having to alt-tab out of the game to do so.

The only problem with this method is that it does largely unnecessary I/O - every time the analyzer checks to see if there is a newer file in the folder, it has to read from the disk at least once for every file in the folder. If I understand the 'access time' mechanism correctly, all of those reads also cause writes to the disk as the operating system updates each file's access time! (Note that this is assuming that the OS is actually set to do so, which I believe is the default. It can be set not to perform this behavior, but changing that is not within my program's rights, nor should it be.)

The idea of performing reads and writes every second just to facilitate a minor convenience has bothered me since I wrote it, but I thought that there was no alternative. Just as I began writing this post, however, I thought of an alternative - instead, I will put the user commands I mentioned in an earlier post to work. If the user wants to start a new logfile and switch to it, they will enter a command that does so.

That in itself poses an interesting problem, though: the user can only issue commands to the application through the logfile that the application is currently processing. The simple act of injecting a command into that logfile will make it the most recently modified file in the folder, meaning that the application will still think that the current logfile is the proper logfile to use. If the user stops logging to that logfile in an effort to keep it from being modified, the program happily continues to sit there waiting for the logfile to update since it hasn't been told to use a new logfile. What to do?

Instead of simply looking for the latest modified file in the directory, the application will instead look for the latest modified file in the directory that is not the currently active logfile. The only problem with this approach is if the user issues the command in error, this will cause the program to grab whatever logfile is 'second in line,' whether it was created 5 seconds ago or 5 months ago. The solution to this problem is to place a limit on how recently a file has to have been changed in order to be considered by the application - say, 30 seconds by default, modifiable by the user.

Displaying the Log

The original idea for how to display the combat log is inefficient compared to another method I've seen. The original idea is as follows:

As the normal combat log is written to disk by LOTRO, the parser continuously checks the file to see if its modification time has changed. If the time has changed, then the parser grabs a 'chunk' from the file, pulling every line it finds and then setting its internal seek position to the current end of the file. On the next 'chunk check,' the process is repeated, and so on.

I've been wanting to find a better way to do this for quite some time now. I feel uncomfortable running the 'chunk check' any more than 4 times per second because of disk I/O concerns like with the last issue, but for the finest precision of calculations, the application would ideally be checking for a new line at least 10 times a second.

I checked the source code for cstats, specifically its LogReader.cs file, to see how that program does it. If I'm reading it right, it actually checks the logfile every millisecond for a new line, and in a more efficient way than I do it.

My program first asks if there's been a change, opens the file if there has been, and gets the chunk.

CStats just does a readline on some kind of file stream - I don't really understand streams very well yet, but I intend to use the same kind of process in my application now that I've seen how CStats does it. Time to stop writing and start researching!