Modifying Pyke
Index to This Page
Mercurial Repositories
With Mercurial, you clone the entire repository locally on your computer. Then you can make changes and commit those changes to your local repository. If you think those changes might be interesting to everybody, make your local repository (or a clone of it) publicly available (either on your own server, or on one of the Mercurial Hosting Sites) and send me an email. I will pull your changes, examine them, and push them to the master repository on sourceforge.
Mercurial Keyword Extension
The Pyke sources use the Mercurial Keyword Extension as a holdover from when the repository used Subversion rather than Mercurial.
The hgrc_keywords file has been provided to enable and configure this extension for Pyke use. You can append this file to either your personal .hgrc configuration file (which would then apply to all of your Mercurial projects) or the project .hg/hgrc file (see hgrc in the Mercurial wiki).
If you use a post-clone Mercurial hook, or append hgrc_keywords manually after cloning, the keywords won't be expanded properly when the project is first cloned. But they will be expanded properly if the clone is done with the -U option and then an hg update done in the newly cloned repository (after the changes to .hg/hgrc have been made).
The keyword expansions are only used by the tools that generate the html documentation (see Rebuilding the HTML Documentation, below).
Which Repository Do I Use?
Normally, you will clone one of the following four repositories locally to make a master copy of what's on sourceforge. Then you would clone your master copy (which is very fast) to make separate clones for each development task that you are working on for Pyke.
So it is best to keep all of these clones together in a common directory.
There are four repositories on sourceforge that you can start with:
- release_1
- Use this for bug fixes, code and documentation cleanup, and anything else that would go into a point release for release 1. I merge the changes made here into all of the other repositories. So this code goes into both the Python2.x and Python3.x versions of Pyke.
- pyke
- Use this for major new features that would result in a major new release (e.g., release 1.2). I merge the changes made in release_1 into the pyke repository (but maybe not the other way around). And I merge the changes made in the pyke repository into the pre_2to3 repository. So the code here goes into both the Python2.x and Python3.x future versions of Pyke.
- pre_2to3_r1
Use this for bug fixes, code and documentation cleanup, and anything else that would go into a point release for release 1, but only apply to the Python3.x version of Pyke. I merge the changes made in release_1 into the pre_2to3_r1 repository (but not the other way around). And I merge the changes made in the pre_2to3_r1 repository into the pre_2to3 repository. So changes here only go into the next point release of the Python3.x version of Pyke.
Warning
This code is maintained in a state just prior to running Python's 2to3 tool on it. So you can't just run the code here directly.
The run_2to3 script runs 2to3 on the current copy of the sources. Do not run this in a repository clone that you still want to use to do commits! Instead, commit all of your changes, then clone the repository and do run_2to3 in the clone. If anything doesn't work, go back to the first repository to fix it, delete the clone, and repeat the whole process. This was done to minimize merge conflicts caused by the 2to3 changes.
The run_pre_test script will:
- clone the current repository
- then in the clone do:
- run_2to3
- testpyke -3.1
- python setup.py -q sdist --formats zip
- insert '3' after 'pyke' in the name of the source distribution zip file.
Run_pre_test assumes that you either have the keywording options set in your personal .hgrc file, or have clone hooks in place to copy these into the .hg/hgrc file of all clones within your pyke work area. See Mercurial Keyword Extension, above.
- pre_2to3
- Normally I merge changes from the pyke repository and the pre_2to3_r1 repository into pre_2to3 so that nothing needs to be done in this repository. Most major new features would be developed in the pyke repository and merged into pre_2to3. Making changes to pre_2to3 directly would only be done when those changes are for major new features that only apply to the Python3.x version of Pyke.
So, for example, if you wanted to work on the release_1 repository, you would:
$ mkdir pyke_repos $ cd pyke_repos $ hg clone -U http://pyke.hg.sourceforge.net:8000/hgroot/pyke/release_1 master $ hg clone master task_1 $ cd task_1
Note
This assumes that you've added the hgrc_keywords file to your ~/.hgrc file. See Mercurial Keyword Extension, above.
Compiling PLY Tables Files
Pyke uses PLY (Python Lex and Yacc) as it's parser generator. PLY compiles the Pyke grammars into a set of three tables files:
- kfbparser_tables.py (from kfbparser.py)
- krbparser_tables.py (from krbparser.py)
- scanner_tables.py (from scanner.py)
A copy of PLY is included in the source directory (pyke/krb_compiler/ply) so that there there can be no version mismatch between the version of PLY used to compile these tables files and the version of PLY installed on your machine.
To regenerate these tables files, at the top-level source directory:
$ python >>> from pyke.krb_compiler import kfbparser, krbparser, scanner >>> scanner.init(scanner, 0, True) >>> krbparser.init(krbparser, True) >>> kfbparser.init(kfbparser, True)
or just run the "testall.py" program from the doctest-tools package:
$ cd pyke/krb_compiler $ testall.py
Compiling the Compiler.krb File
Pyke uses itself to compile your rule base sources (.krb files) into Python source (.py) files.
The knowledge base file that Pyke uses for this is pyke/krb_compiler/compiler.krb. This gets compiled into compiler_bc.py, which is stored in the source code repository.
To recompile the compiler_bc.py file, from the top-level source directory:
$ mkdir pyke/krb_compiler/compiled_krb
$ python
>>> from pyke import krb_compiler
>>> krb_compiler.compile_krb('compiler', 'pyke.krb_compiler.compiled_krb',
... 'pyke/krb_compiler/compiled_krb',
... 'pyke/krb_compiler/compiler.krb')
['compiler_bc.py']
$ mv pyke/krb_compiler/compiled_krb/compiler_bc.py pyke/krb_compiler
Running Unit Tests
The doctest-tools package is required to run the unit tests (see Other Required Packages for more details).
The testall.py and testdoc.py scripts from doctest-tools can be run anywhere.
In addition, the top-level directory contains a testpyke script that will delete all of the compiled_krb directories, then run testall.py twice. The first run must recompile all of the knowledge base sources (.krb, .kfb and .kqb files) into the compiled_krb directories in order to run the tests. The second run reuses the files compiled in the first run. This makes sure that all of the tests run properly whether they have to compile the knowledge base sources or not.
Rebuilding the HTML Documentation
The doc/html directory contains all of the documents that you are reading now. These are ready to browse directly from your hard drive if you'd like.
The documentation is generated using the rest2web package, which uses docutils (see Other Required Packages for more details).
The sources for the documentation are in doc/source. Each .txt file there is converted into an .html file in the doc/html directory by running:
$ cd doc/source $ bin/gen_html
This takes about 9 seconds. It:
- Temporarily appends hyperlink references onto all of the *.txt files.
- Runs r2w to regenerate the files in doc/html
- except for those in doc/html/stylesheets and doc/html/images.
- Strips all of the hyperlink references from the *.txt files.
- Creates a new sitemap.xml file with all of the dates that the files were last modified.
Note
This process uses the date information expanded by the Mercurial Keyword Extension. See Mercurial Keyword Extension, above.
I've gone ahead and placed the generated html files in the source repository so that you can browse the documentation locally without having to run bin/gen_html. So you only need these procedures if you change the documentation (i.e., change the .txt files in doc/source).
To test all of the code examples in the documents, use the testall.py command from the doctest-tools package:
$ cd doc/source $ testall.py