revised README: cropped images, refactored sample code, added examples and custom source group setup

This commit is contained in:
Eberhard Graether
2018-12-19 13:12:32 +01:00
parent 807b80a1fd
commit 353e26d47e
17 changed files with 190 additions and 87 deletions
+2 -2
View File
@@ -1,7 +1,7 @@
# Changelog
## v1_db23_p0
## v1.db23.p0
**2018-12-18**
* This is the first official release of the SourcetrailDB project.
* First official release of the SourcetrailDB project.
+185 -82
View File
@@ -1,58 +1,131 @@
# SourcetrailDB
Library to export Sourcetrail compatible database files for writing custom indexers
Library to export [Sourcetrail](https://www.sourcetrail.com/) compatible database files for writing custom indexers
## CI Pipelines
Windows Builds: [![Build status](https://ci.appveyor.com/api/projects/status/gbblx1l790vm952c/branch/master?svg=true)](https://ci.appveyor.com/project/mlangkabel/sourcetraildb/branch/master)
Windows: [![Build status](https://ci.appveyor.com/api/projects/status/gbblx1l790vm952c/branch/master?svg=true)](https://ci.appveyor.com/project/mlangkabel/sourcetraildb/branch/master)
Linux and macOS: [![Build Status](https://travis-ci.org/CoatiSoftware/SourcetrailDB.svg?branch=master)](https://travis-ci.org/CoatiSoftware/SourcetrailDB)
Linux and Mac Builds: [![Build Status](https://travis-ci.org/CoatiSoftware/SourcetrailDB.svg?branch=master)](https://travis-ci.org/CoatiSoftware/SourcetrailDB)
## Introduction
The SourcetrailDB project provides write access to [Sourcetrail](https://www.sourcetrail.com/) database files. This means that you can use the SourcetrailDB project to write an indexer for an arbitrary programming language (or [other kind of data](https://github.com/CoatiSoftware/SourcetrailDB/tree/master/examples/cpp_poetry_indexer)) and make this indexer output a Sourcetrail database file that can be viewed and navigated via the Sourcetrail tool.
The SourcetrailDB project provides write access to [Sourcetrail](https://www.sourcetrail.com/) database files. You can use the SourcetrailDB project to write an indexer for an arbitrary programming language (or other kind of data, e.g. see [poetry indexer example](examples/cpp_poetry_indexer)) and export a Sourcetrail database file that can be viewed and navigated within Sourcetrail.
## Projects Using the SourcetrailDB
## Projects Using SourcetrailDB
* [SourcetrailPythonIndexer](https://github.com/CoatiSoftware/SourcetrailPythonIndexer)
## Supported Languages
## Supported Language Bindings
Note that even though the core of this project is written in C++, this does __not__ require you to write your indexer in C++ as well. Instead you can use the SourcetrailDB via one of our bindings for other languages. Currently bindings for the following languages are available:
Even though the core implementation is written in C++, this does not require you to write your indexer in C++ as well. Instead you can use a language binding (e.g. see [SWIG](http://www.swig.org/)). These language bindings are already available:
* Python (via [SWIG](http://www.swig.org/))
If the language of your choice is not covered by this list, feel free to open an issue on the [issue tracker](https://github.com/CoatiSoftware/SourcetrailDB/issues) or even better: join us in our efforts on this project and provide a pull request!
If the language of your choice is not covered by this list, feel free to [open an issue](https://github.com/CoatiSoftware/SourcetrailDB/issues) or provide a pull request.
## Versioning
The SourcetrailDB version format consists of three numbers and looks like this `vXX_dbYY_pZZ`.
* `XX` marks the interface version of the API. This version is increases on every change that breaks backwards compatibility of the API.
* `YY` marks the version of the database that will be generated when using the SourcetrailDB API. This version needs to match the database version of the Sourcetrail instance that is used to open the generated database file to be compatible.
* `ZZ` marks the patch number of the build. It will increase with every release that publishes bugfixes and features that don't break any compatibility.
The SourcetrailDB version format consists of three numbers in the format `vXX.dbYY.pZZ`.
* `XX`: interface version. This version increases on every change that breaks backwards compatibility.
* `YY`: Sourcetrail database version. This version needs to match the database version of the used Sourcetrail instance. You can find the database version of Sourcetrail in its About dialog.
* `ZZ`: patch number of the build. It will increase with every release that publishes bugfixes and features that don't break any compatibility.
You can find a complete list of available releases on the [GitHub release page](https://github.com/CoatiSoftware/SourcetrailDB/releases). If you are interested in the actual changes of every release, please take a look at the [Changelog](https://github.com/CoatiSoftware/SourcetrailDB/blob/master/CHANGELOG.md).
You can find a complete list of available releases on the [GitHub release page](https://github.com/CoatiSoftware/SourcetrailDB/releases). If you are interested in the actual changes of every release, please take a look at the [Changelog](CHANGELOG.md).
## Writing to the Sourcetrail Database
## Building
The SourcetrailDB tries to keep the API for writing to the Sourcetrail database as simple as possible. For the C++ core take a look at the list of methods of the [SourcetrailDBWriter](https://github.com/CoatiSoftware/SourcetrailDB/blob/master/core/include/SourcetrailDBWriter.h) class for a complete overview of the provided write API. If you are using bindings for other languages, the code may look different but the set of provided functionality should stay the same. For now, let's jump right into action with some examples!
Take a look at [Appveyor (for Windows)](appveyor.yml) or [Travis (for Linux and macOs)](.travis.yml) CI scripts for reference.
### SourcetrailDB Core
Requirements:
* [CMake](https://cmake.org/) >= 2.6
* C++-Compiler with C++11 support
To build the SourcetrailDB core library `libsourcetraildb.a`:
```
$ cd path/to/SourcetrailDB
$ mkdir build
$ cd build
$ cmake ..
$ make lib_core
```
To run the tests, just build and execute the test executable:
```
$ make test_core
$ ./core/test_core
```
### Python Bindings
Requirements:
* [Python](https://www.python.org/) needs to be included and linked against when building the Python bindings. CMake will auto-detect your Python installation. If you want to build against a specific version of Python, please define the `PYTHON_LIBRARY` variable accordingly when running CMake. Make sure to link to the correct Python version when building for different architectures (32bit/64bit).
* [SWIG 3.0.12](http://www.swig.org/) is used to automatically generate Python binding code. Make sure that SWIG is added to your path environment variable.
If you want to build the Python bindings run:
```
$ cd path/to/SourcetrailDB
$ mkdir build
$ cd build
$ cmake -DBUILD_BINDINGS_PYTHON=ON ..
$ make _sourcetraildb
```
Swig is configured to generate the Python binding code as a pre-build event, so you don't need to bother with updating manually.
### Examples
The examples help you to understand SourcetrailDB usage in practice. Please take a look at each examples README file for build and use instructions. Each example also provides a Sourcetrail project file `.srctrlprj` showing you how to use a custom indexer directly from Sourcetrail (see [Integrating with Sourcetrail](#integrating-with-sourcetrail)).
* [C++ API Example](examples/cpp_api_example)
* [C++ Poetry Indexer](examples/cpp_poetry_indexer)
* [Python API Example](examples/python_api_example)
### Recording a Simple Symbol
## SourcetrailDB API
SourcetrailDB tries to keep the API for writing to the Sourcetrail database as simple as possible. For the C++ core take a look at the list of methods of the [SourcetrailDBWriter](core/include/SourcetrailDBWriter.h) class for a complete overview of the provided write API. If you are using bindings for other languages, the code may look different but the set of provided functionality should stay the same.
### Opening or Creating a Database
```c++
// create an instance of SourcetrailDBWriter
sourcetrail::SourcetrailDBWriter writer;
// open a database connection by passing a database file path (.srctrldb or .srctrldb_tmp)
// if the database file does not exist yet, it will be created in this location.
// a project file (.srctrlprj) will also be created, if it does not exist in this location.
writer.open("MyProject.srctrldb");
// record data...
// close the database connection.
writer.close();
```
### Recording a Symbol
```c++
sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
writer.recordSymbol({ "::",{ { "void", "foo", "()" } } }); // don't worry, this will syntax become clear in a minute
// records a named non-indexed symbol without type
writer.recordSymbol({ "::", { { "void", "foo", "()" } } });
writer.close();
```
!["Recording a Simple Symbol"](images/readme/01_recording_a_simple_symbol.png "Recording a Simple Symbol")
!["Recording a Symbol"](images/readme/01_recording_a_simple_symbol.png "Recording a Symbol")
### Recording Additional Symbol Information
@@ -61,16 +134,24 @@ writer.close();
sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
sourcetrail::NameHierarchy name = { "::",{ { "void", "foo", "()" } } };
sourcetrail::NameHierarchy name = { "::", { { "void", "foo", "()" } } };
// the returned symbolId will be used to record further information for the symbol
int symbolId = writer.recordSymbol(name);
assert(symbolId == writer.recordSymbol(name)); // recording a symbol multiple times will always return the same id
writer.recordSymbolDefinitionKind(symbolId, sourcetrail::DEFINITION_EXPLICIT); // makes the "non-indexed" go away
writer.recordSymbolKind(symbolId, sourcetrail::SYMBOL_FUNCTION); // turns the "symbol" into "function"
// recording a unique name multiple times will always return the same symbol id
assert(symbolId == writer.recordSymbol(name));
// set symbol "explicit" to remove the "non-indexed" state.
writer.recordSymbolDefinitionKind(symbolId, sourcetrail::DEFINITION_EXPLICIT);
// set type to turn the symbol into "function"
writer.recordSymbolKind(symbolId, sourcetrail::SYMBOL_FUNCTION);
writer.close();
```
!["Recording additional Symbol Information"](images/readme/02_recording_additional_symbol_information.png "Recording additional Symbol Information")
!["Recording Additional Symbol Information"](images/readme/02_recording_additional_symbol_information.png "Recording Additional Symbol Information")
### Recording Nested Symbol Names
@@ -79,24 +160,30 @@ writer.close();
sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject_3.srctrldb");
sourcetrail::NameHierarchy name; // now lets look into names
name.nameDelimiter = "::"; // in many languages names can be nested - here you specify the delimiter string.
// create a name hierarchy
sourcetrail::NameHierarchy name;
// specify the delimiter string - in many languages names can be nested
name.nameDelimiter = "::";
// add a name element for the parent
sourcetrail::NameElement parentElement;
parentElement.prefix = ""; // the prefix is just shown in the tooltip and can be omitted if not applicable
parentElement.name = "Bar"; // the name is shown in the graph and is prepended to the name of child a element
parentElement.postfix = ""; // the postfix is just shown in the tooltip and can be omitted if not applicable
parentElement.prefix = ""; // only shown in tooltip of the parent symbol
parentElement.name = "Bar";
parentElement.postfix = ""; // only shown in tooltip of the parent symbol
name.nameElements.push_back(parentElement);
// add a name element for the child
sourcetrail::NameElement childElement;
childElement.prefix = "void"; // tooltips omit prefixes of parents and only show the prefix of the hovered element
childElement.prefix = "void"; // only shown in tooltip
childElement.name = "bar";
childElement.postfix = "()"; // tooltips omit postfixes of parents and only show the prefix of the hovered element
childElement.postfix = "()"; // only shown in tooltip
name.nameElements.push_back(childElement);
int symbolId = writer.recordSymbol(name);
writer.recordSymbolDefinitionKind(symbolId, sourcetrail::DEFINITION_EXPLICIT);
writer.recordSymbolKind(symbolId, sourcetrail::SYMBOL_METHOD);
// recording a name hierarchy with multiple elements automatically creates parent symbols, if still unknown
int childId = writer.recordSymbol(name);
writer.recordSymbolDefinitionKind(childId, sourcetrail::DEFINITION_EXPLICIT);
writer.recordSymbolKind(childId, sourcetrail::SYMBOL_METHOD);
writer.close();
```
@@ -104,28 +191,35 @@ writer.close();
!["Recording Nested Symbol Names"](images/readme/03_recording_nested_symbol_names.png "Recording Nested Symbol Names")
### Recording a Symbol's Location
### Recording Symbol Location
```c++
sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
int symbolId = writer.recordSymbol({ "::",{ { "", "Bar", "" },{ "void", "bar", "()" } } });
int symbolId = writer.recordSymbol({ "::", { { "", "Bar", "" }, { "void", "bar", "()" } } });
int fileId = writer.recordFile("C:/example/Bar.cpp"); // path to the source file that contains the recorded location
// record a file via it's absolute path
int fileId = writer.recordFile("C:/example/Bar.cpp");
// create a location
sourcetrail::SourceRange location;
location.fileId = fileId;
location.startLine = 8; // recorded lines start at 1, not 0
location.startColumn = 7; // recorded columns start at 1, not 0
location.startLine = 8; // lines start at 1, not 0
location.startColumn = 7; // columns start at 1, not 0
location.endLine = 8;
location.endColumn = 9;
writer.recordSymbolLocation(symbolId, location); // lets you click symbols in Sourcetrail's code view
// adds a clickable location to the file for the specified symbol
writer.recordSymbolLocation(symbolId, location);
// the whole scope location will be visible and highlighted when the symbol is active
writer.recordSymbolScopeLocation(symbolId, { fileId, 8, 1, 11, 1 });
writer.close();
```
!["Recording a Symbol's Location"](images/readme/04_recording_a_symbols_location.png "Recording a Symbol's Location")
!["Recording Symbol Location"](images/readme/04_recording_a_symbols_location.png "Recording Symbol Location")
### Recording References Between Symbols
@@ -134,11 +228,15 @@ writer.close();
sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
int contextSymbolId = writer.recordSymbol({ "::",{ { "", "Bar", "" },{ "void", "bar", "()" } } });
int referencedSymbolId = writer.recordSymbol({ "::",{ { "void", "foo", "()" } } });
int referenceId = writer.recordReference(contextSymbolId, referencedSymbolId, sourcetrail::REFERENCE_CALL); // edges always go from the context to the referenced symbol
int contextSymbolId = writer.recordSymbol({ "::", { { "", "Bar", "" }, { "void", "bar", "()" } } });
int referencedSymbolId = writer.recordSymbol({ "::", { { "void", "foo", "()" } } });
// edges always go from the context to the referenced symbol
int referenceId = writer.recordReference(contextSymbolId, referencedSymbolId, sourcetrail::REFERENCE_CALL);
// add a location to the reference - highlights the location in the code view when clicking the edge in the graph view
int fileId = writer.recordFile("C:/example/Bar.cpp");
writer.recordReferenceLocation(referenceId, { fileId, 10, 3, 10, 5 }); // clicking an edge will highlight this locaion in the code view
writer.recordReferenceLocation(referenceId, { fileId, 10, 3, 10, 5 });
writer.close();
```
@@ -153,7 +251,9 @@ sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
int fileId = writer.recordFile("C:/example/Bar.cpp");
writer.recordFileLanguage(fileId, "cpp"); // this enables syntax highlighting for predefined languages
// enable syntax highlighting using a <language>.rules file loaded from Sourcetrail's data directory
writer.recordFileLanguage(fileId, "cpp");
writer.close();
```
@@ -161,21 +261,22 @@ writer.close();
!["Recording File Information for Syntax Highlighting"](images/readme/06_recording_file_information_for_syntax_highlighting.png "Recording File Information for Syntax Highlighting")
### Recording Occurrences of Local Symbols
### Recording Local Symbols
```c++
sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
int id = writer.recordLocalSymbol("some_unique_name"); // this name is just for referencing the symbol and won't be displayed anywhere
// provide a unique local symbol name - the name is not displayed anywhere
int localId = writer.recordLocalSymbol("some_unique_name");
int fileId = writer.recordFile("C:/example/Foo.cpp");
writer.recordLocalSymbolLocation(id, { fileId, 3, 6, 3, 6 });
writer.recordLocalSymbolLocation(id, { fileId, 4, 2, 4, 2 });
writer.recordLocalSymbolLocation(localId, { fileId, 3, 6, 3, 6 });
writer.recordLocalSymbolLocation(localId, { fileId, 4, 2, 4, 2 });
writer.close();
```
!["Recording Occurrences of Local Symbols"](images/readme/07_recording_occurrences_of_local_symbols.png "Recording Occurrences of Local Symbols")
!["Recording Local Symbols"](images/readme/07_recording_occurrences_of_local_symbols.png "Recording Local Symbols")
### Recording Comments
@@ -185,7 +286,9 @@ sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
int fileId = writer.recordFile("C:/example/Bar.cpp");
int id = writer.recordCommentLocation({ fileId, 3, 2, 7, 4 }); // causes Sourcetrail to treat the source range as atomic - either display it completely or don't show it at all
// the recorded source range is atomic - it is always shown completely
int id = writer.recordCommentLocation({ fileId, 3, 2, 7, 4 });
writer.close();
```
@@ -200,10 +303,12 @@ sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
int fileId = writer.recordFile("C:/example/Foo.cpp");
// store and show parsing and indexing errors
std::string message = "Really? You missed that \";\" again?";
bool fatal = false;
sourcetrail::SourceRange location = { fileId, 4, 4, 4, 4 };
int id = writer.recordError(message, fatal, location); // store and show parsing and indexing errors
int id = writer.recordError(message, fatal, location);
writer.close();
```
@@ -217,53 +322,51 @@ writer.close();
sourcetrail::SourcetrailDBWriter writer;
writer.open("MyProject.srctrldb");
writer.beginTransaction(); // wrapping your code in transactions really speeds up database operations
// wrapping your code in transactions really speeds up database operations
writer.beginTransaction();
for (int i = 0; i < 1000; ++i)
{
int id = writer.recordSymbol({ "::", { { "void", "foo" + std::to_string(i), "()" } } });
if (id == 0) // something must have gone wrong because 0 is not a valid id
if (id == 0) // if something goes wrong
{
writer.rollbackTransaction(); // discard all the changes made within the transaction
// discard all the changes made within the transaction
writer.rollbackTransaction();
writer.close();
return;
}
}
writer.commitTransaction(); // don't forget to commit your transaction to persistently write the changes
// don't forget to commit your transaction to persistently write the changes
writer.commitTransaction();
writer.close();
```
## Integrating with Sourcetrail
## Building the SourcetrailDB Project
Applications using SourcetrailDB can be directly integrated with Sourcetrail by creating a project with a **Custom Command Source Group**. Choose `Custom` in the project selection dialog:
If you are unsure, just look at the [Appveyor (for Windows)](https://github.com/CoatiSoftware/SourcetrailDB/blob/master/appveyor.yml) or [Travis (for Mac and Linux)](https://github.com/CoatiSoftware/SourcetrailDB/blob/master/.travis.yml) CI scripts. But in general building the SourcetrailDB project is quite simple.
<img src="images/readme/10_custom_command_source_group_selection.png" alt="Custom Command Source Group Selection" width="720"/>
### Project Setup
A Custom Command Source Group defines:
* **Custom Command**: This command will be executed for each source file in the source group. You can pass these variables in the command:
- **%{SOURCE_FILE_PATH}**: The path to each source file in the source group
- **%{DATABASE_FILE_PATH}**: The path to the database of the project
- **%{DATABASE_VERSION}**: The database version of the used Sourcetrail instance
- **%{PROJECT_FILE_PATH}**: The path to the project file
* **Files & Directories to Index**: Paths to search for source files.
* **Excluded Files & Directories**: Files and directories within the above, that should not be indexed.
* **Source File Extensions**: The extension of the source files. Extension is ignored if left empty.
!["Custom Command Source Group Setup"](images/readme/11_custom_command_source_group_setup.png "Custom Command Source Group Setup")
### The Core Target
### Indexing
The core project does not have any external dependencies you need to install, so just check out the project and run
```
$ mkdir build
$ cd build
$ cmake -G "<Your Generator>" ..
```
Indexing the project works as usual, the specified **Custom Command** is executed with each source file passed in **%{SOURCE_FILE_PATH}**. The files in the **Custom Command Source Group** will be indexed after all other files.
This will generate some build instructions that let you build the core library target and the test target. To run the tests, just execute the built executable.
### The Python Bindings Target
This target requires you to have the following software installed before configuring your CMake:
* [Python](https://www.python.org/) needs to be included and linked against when building the Python bindings. Usually CMake will auto-detect your Python installation. If you want to build against a specific version of Python, please define the `PYTHON_LIBRARY` variable accordingly when running CMake. Make sure to link to a 32 bit version of Python when building this target for a 32 bit architecture and use a 64 bit Python when bulding this target for a 64 bit architecture.
* [SWIG 3.0.12](http://www.swig.org/) is used to automatically generate Python binding code. Make sure that SWIG is added to your path environment variable.
If you want to build the Python bindings run:
```
$ mkdir build
$ cd build
$ cmake -G "<Your Generator>" -DBUILD_BINDINGS_PYTHON=ON ..
```
Swig is configured to generate the Python binding code as a pre-build event, so you don't need to bother with updating that part manually.
<img src="images/readme/12_custom_command_source_group_indexing.png" alt="Custom Command Source Group Indexing" width="343"/>
@@ -2,7 +2,7 @@
<config>
<source_groups>
<source_group_78efd997-84bf-4db6-9736-f5ddd702053c>
<custom_command>./cpp_api_example $DB_PATH $SOURCE_PATH $DB_VERSION</custom_command>
<custom_command>./cpp_api_example %{DATABASE_FILE_PATH} %{SOURCE_FILE_PATH} %{DATABASE_VERSION}</custom_command>
<name>Custom Command Source Group</name>
<source_extensions>
<source_extension>.cpp</source_extension>
@@ -2,7 +2,7 @@
<config>
<source_groups>
<source_group_78efd997-84bf-4db6-9736-f5ddd702053c>
<custom_command>./poetry_indexer $DB_PATH $SOURCE_PATH $DB_VERSION</custom_command>
<custom_command>./poetry_indexer %{DATABASE_FILE_PATH} %{SOURCE_FILE_PATH} %{DATABASE_VERSION}</custom_command>
<exclude_filters>
<exclude_filter>@POETRY_INDEXER_DATA_PATH@/COPYRIGHT.txt</exclude_filter>
</exclude_filters>
@@ -2,7 +2,7 @@
<config>
<source_groups>
<source_group_78efd997-84bf-4db6-9736-f5ddd702053c>
<custom_command>python example.py --database-file-path=$DB_PATH --source-file-path=$SOURCE_PATH</custom_command>
<custom_command>python example.py --database-file-path=%{DATABASE_FILE_PATH} --source-file-path=%{SOURCE_FILE_PATH}</custom_command>
<name>Custom Command Source Group</name>
<source_extensions>
<source_extension>.py</source_extension>
Binary file not shown.

Before

Width:  |  Height:  |  Size: 61 KiB

After

Width:  |  Height:  |  Size: 46 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 58 KiB

After

Width:  |  Height:  |  Size: 44 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 69 KiB

After

Width:  |  Height:  |  Size: 55 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 79 KiB

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 75 KiB

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 71 KiB

After

Width:  |  Height:  |  Size: 57 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 66 KiB

After

Width:  |  Height:  |  Size: 52 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 78 KiB

After

Width:  |  Height:  |  Size: 65 KiB

Binary file not shown.

Before

Width:  |  Height:  |  Size: 64 KiB

After

Width:  |  Height:  |  Size: 62 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 343 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 372 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 55 KiB