code_tokenize v0.1.0

First main release of code.tokenize

First version to extend the functionality of the underlying AST parser.

Changes

• tokenize parses source code now with language specific configuration
• For Python, we automatically detect indentations and add special tokens
• Code is now extensively tested in parsing large libraries (Python and Java)
• Update documentation to make usage more easier

Minor features (still under test)

• AST path based detection of token types (detection of variable usages, definitions or function calls)
• Language specific configuration for Java