Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
220 commits
Select commit Hold shift + click to select a range
9467fbb
Update README.md file
svdhoog Feb 8, 2018
ea4305d
Update README.md
svdhoog Feb 8, 2018
8ff1eab
Update README.md
svdhoog Feb 8, 2018
9c5b837
Update README.md
svdhoog Feb 8, 2018
fdf7a9a
AUTHORS and LICENSE files
svdhoog Feb 8, 2018
22a488f
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Feb 8, 2018
a5b6783
AUTHORS and LICENSE files
svdhoog Feb 8, 2018
7ac3518
Manual FLAViz initial commit
svdhoog Feb 15, 2018
02653bd
Readme file for FLAVis
svdhoog Mar 5, 2018
dbc6bdf
Started to create a valid structure and reproduced plots from gitpage
0xfabi Apr 9, 2018
f95d85a
updated documentation inside /doc
0xfabi Apr 10, 2018
c70f540
Cleaned up project structure
0xfabi Apr 10, 2018
8fe3584
Added relative paths for input and output
0xfabi Apr 12, 2018
92e33cb
Added possibilty to change between absolute or relative path in confi…
0xfabi Apr 12, 2018
27a6db7
Minor changes in parameter.rst for the new paths
0xfabi Apr 12, 2018
56719bc
Merge pull request #1 from 0xfabi/master
svdhoog Apr 24, 2018
e690b5f
Updated data conversion scripts; added gendb.py and gendb_special.py
svdhoog Apr 24, 2018
908ac77
Updated readme files for replace_xml_function script
svdhoog Apr 24, 2018
6caedea
Updated readme file for FLAViz root folder
svdhoog Apr 24, 2018
6810f7d
Updated readme
svdhoog Apr 25, 2018
c16ab5f
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Apr 25, 2018
a95ff50
Update readme
svdhoog Apr 25, 2018
e34277f
Update readme
svdhoog Apr 25, 2018
975b54e
Merge pull request #1 from svdhoog/master
0xfabi Apr 26, 2018
e66be83
Cleaned up conversion scripts structure
0xfabi Apr 26, 2018
b0ea9e4
Cleaned up conversion scripts structure2
0xfabi Apr 26, 2018
8129ed0
Made some changes.
0xfabi Apr 26, 2018
598f4a7
Renamed xml writeout function
0xfabi Apr 26, 2018
ff25dc5
Updated docs xml
0xfabi Apr 26, 2018
021eaa4
Updated manual
0xfabi Apr 26, 2018
d696161
Cleaned up TODO folder
0xfabi Apr 26, 2018
4fa4c6a
Updated manual with workflow diagramms
0xfabi Apr 26, 2018
63df116
Merge pull request #3 from 0xfabi/master
svdhoog Apr 26, 2018
676e72b
Updated gitignore
0xfabi Apr 26, 2018
97b43f2
gendb.py executeable
0xfabi Apr 26, 2018
ffc97c2
Ported gendb to py3
0xfabi Apr 26, 2018
e719f57
Edited documentation and corrected paths
0xfabi Apr 26, 2018
6178816
Update README.md
svdhoog May 3, 2018
0503722
Update README.md
svdhoog May 3, 2018
ae739b7
Added verbose mode
0xfabi May 3, 2018
908617e
Changed parameterpath to configpath and updated docu
0xfabi May 3, 2018
4aaadd0
Changed parameterpath to configpath and updated docu
0xfabi May 3, 2018
b81d5ff
Added progressbar for preprocessing and visualisation
0xfabi May 3, 2018
6001bbb
Updated progressbar steps
0xfabi May 8, 2018
96a1f4a
Fixed progressbar. Is now printed in one line for each step.
0xfabi May 8, 2018
673746c
Update README.md
svdhoog May 8, 2018
c22d3f0
Update README.md
svdhoog May 8, 2018
6e9c367
Update README.md
svdhoog May 8, 2018
555f978
Added repo_name to config.yaml.
0xfabi May 8, 2018
9d5435b
Update README.md
svdhoog May 8, 2018
a786606
Updated documentation
0xfabi May 8, 2018
fa44f0a
Updated README.md
0xfabi May 8, 2018
a6f8d85
Merge branch 'master' into master
svdhoog May 8, 2018
e4d492f
Color fix for timeseries
0xfabi May 10, 2018
d14ff82
Merge branch 'master' of https://github.com/0xfabi/ETACE
0xfabi May 10, 2018
bf0c6cc
Fixed random colored plots
0xfabi May 10, 2018
ef81754
fixed gendb_special.py and ported to py3
0xfabi May 10, 2018
e494369
Add argparser for all files including outputpath
0xfabi May 10, 2018
89559ee
Fixed gendb
0xfabi May 10, 2018
e4c62e9
Fixed gendb
0xfabi May 10, 2018
7609d33
Added .db to gitignore
0xfabi May 10, 2018
007234a
Updated README
0xfabi May 10, 2018
85e0da5
Fixed type in README
0xfabi May 10, 2018
3f8716d
Added possiblility to change name of iters.db
0xfabi May 11, 2018
1073ac5
Added better -h description for gendb
0xfabi May 11, 2018
fcb2e50
Added full usage example for gendb in usage_help.rst
0xfabi May 11, 2018
ccd422f
Updated gendb and doc
0xfabi May 11, 2018
bd94ee0
Update README.md
svdhoog May 11, 2018
cc0a8cf
Fixed gendb and added check if output directory exists
0xfabi May 11, 2018
ca929f0
Merge branch 'master' into master
0xfabi May 11, 2018
64ee6e5
Updated progressbar
0xfabi May 11, 2018
c0d9d77
Updated progressbar
0xfabi May 11, 2018
216d14f
Merge branch 'master' of https://github.com/0xfabi/ETACE
0xfabi May 11, 2018
8a42c6d
Updated README
0xfabi May 11, 2018
fd8d29d
Added greyscale support
0xfabi May 12, 2018
0ed2b13
Fixed db_hdf5: replaced deprecatedlib
0xfabi May 12, 2018
039f7f1
Added greycolor support for boxplots
0xfabi May 22, 2018
f74e471
Fixed output for db_hdf5
0xfabi May 23, 2018
d3bc3c8
Fixed output for db_hdf5
0xfabi May 23, 2018
15ff30f
Update and rename README.rst to README.md
svdhoog May 24, 2018
fba3d85
Update usage_help.rst
svdhoog May 24, 2018
0d2817d
Update README.md
svdhoog May 24, 2018
b8861ba
Update and rename readme.txt to readme.md
svdhoog May 24, 2018
f1b33fd
Update readme.md
svdhoog May 24, 2018
b7b4f4f
Update readme.md
svdhoog May 24, 2018
3c3a438
Update and rename tutorial.rst to readme.md
svdhoog May 24, 2018
eb6b0b1
Update and rename configurations.rst to configurations.md
svdhoog May 24, 2018
a672a43
Rename configurations.md to config.md
svdhoog May 24, 2018
4e832c0
Update readme.md
svdhoog May 24, 2018
835a83c
Update and rename usage_help.rst to FAQ.md
svdhoog May 24, 2018
c2555d2
Update README.md
svdhoog May 24, 2018
43a7a90
Delete dataset.rst
svdhoog May 24, 2018
62a5f99
Rename readme.md to readme.rst
svdhoog May 24, 2018
dd9e42a
Rename config.md to config.rst
svdhoog May 24, 2018
2209985
Update readme.rst
svdhoog May 24, 2018
f1ea143
Update readme.rst
svdhoog May 24, 2018
94f3124
Update config.rst
svdhoog May 24, 2018
8c11eb9
Update config.rst
svdhoog May 24, 2018
3a6c353
Update readme.rst
svdhoog May 24, 2018
b042ace
Update config.rst
svdhoog May 24, 2018
c236cd9
Combined Step3 and 4 in visualisation output
0xfabi May 24, 2018
cfa0532
Merge branch 'master' of https://github.com/0xfabi/ETACE
0xfabi May 24, 2018
25803c6
multiple_batch possible with var1:x var2:y
0xfabi May 24, 2018
a670370
Set multiple parameters in var1
0xfabi May 24, 2018
a55f66a
Update README.md
svdhoog May 25, 2018
c01daeb
Update README.md
svdhoog May 25, 2018
8f3c4fd
Update README.md
svdhoog May 25, 2018
2decd39
FLAViz Manual tex updated
svdhoog May 25, 2018
72ccbc9
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog May 25, 2018
eff69fa
FLAViz Manual PDF added
svdhoog May 25, 2018
614028a
Update README.md
svdhoog May 25, 2018
01aed47
reolved merge conflicts
svdhoog May 25, 2018
18d04b6
Update config.rst
svdhoog May 26, 2018
e7454d4
Update config.rst
svdhoog May 26, 2018
3cb84e6
Update config.rst
svdhoog May 26, 2018
de22f41
Corrected minor typos in comments in code
svdhoog May 26, 2018
e73ad20
Update config.rst
svdhoog May 27, 2018
7525dfb
Manual.tex corrections
svdhoog May 27, 2018
4f4692f
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog May 27, 2018
0885c2f
Update config.rst
svdhoog May 29, 2018
5279e98
Fixed clr for timeseries many output
0xfabi May 29, 2018
e015b94
Update config.rst
svdhoog May 29, 2018
15150a6
Update README.md
svdhoog May 29, 2018
3e3b06b
Update README.md
svdhoog May 29, 2018
907b41f
Update README.md
svdhoog May 29, 2018
8d27961
Update README.md
svdhoog May 29, 2018
0c2e716
Update readme.rst
svdhoog May 29, 2018
7af8486
Update readme.rst
svdhoog May 29, 2018
8bc62ea
Update README.md
svdhoog May 29, 2018
62da77a
Fixed empty path for merge_hdf_agentwise
0xfabi Jun 4, 2018
ae83ff0
merged
0xfabi Jun 5, 2018
c5ac5d4
merged2
0xfabi Jun 5, 2018
c2a6dd8
Merge branch 'svdhoog-master'
0xfabi Jun 5, 2018
0b1b48a
Renamed and fixed fill_between option from ticket #18
0xfabi Jun 5, 2018
5277d70
Updated docs for visualisation
0xfabi Jun 5, 2018
e2e4628
Updates to example config files
svdhoog Jun 6, 2018
87dfbf0
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Jun 6, 2018
86a1263
Issue #25: Example C code for growth rate transformations
svdhoog Jun 6, 2018
6d342e5
Fixed renaming from ticket #18
0xfabi Jun 6, 2018
53be79e
Added option to create pdf and new names for timeseries plots
0xfabi Jun 7, 2018
7e68e6b
Added option to create pdf and new names for timeseries plots
0xfabi Jun 7, 2018
b189b3a
Changed plot names for histogram
0xfabi Jun 7, 2018
c344b66
Changed plot names for histogram
0xfabi Jun 7, 2018
ac9d7c8
Changed name of scatterplot
0xfabi Jun 7, 2018
81f448d
Changed name of scatterplot
0xfabi Jun 7, 2018
bfb9153
Names for all plots have a new scheme now.
0xfabi Jun 7, 2018
e91dbd0
Removed old plot_name
0xfabi Jun 7, 2018
079eedd
Updated readme for plot_format and removed plot_name
0xfabi Jun 7, 2018
e254fa9
Fixed bug in merge_hdf_agentwise from ticket #23
0xfabi Jun 8, 2018
5324933
Set new root path in config for FLAViz
0xfabi Jun 8, 2018
fced164
Issue #24: New results plots
svdhoog Jun 8, 2018
f7aa50d
Merged with Sander
Jun 15, 2018
224cbb4
Merge branch 'svdhoog-master'
Jun 15, 2018
4b75a85
Added plot_name as a parameter again.
Jun 15, 2018
bf1b840
Added plot_name as a parameter again.
Jun 15, 2018
a9d3234
Updated config for plot_name parameter
Jun 15, 2018
9adb61f
Added py3 convert
Jun 15, 2018
246347c
Merged
Jun 15, 2018
701f0e6
Merge branch 'svdhoog-master'
Jun 15, 2018
35c2167
Bugfixes readme files: Renamed plot option fill_between_quartiles to …
svdhoog Jul 19, 2018
0ecf678
Update FAQ.md
svdhoog Jul 20, 2018
33260a3
Updates transform.py
svdhoog Jul 20, 2018
d2aa4b7
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Jul 20, 2018
4b377a3
Merge pull request #30 from 0xfabi/master
svdhoog Jul 27, 2018
02a59f6
plot.py: commented out ggplot
svdhoog Jul 27, 2018
583dac4
parameters.py: default no. of bins 50
svdhoog Jul 27, 2018
06ef97c
Update readme with info how to contribute
svdhoog Jan 15, 2019
96f7adf
Updated plot_config.py: Plot settings: axes limits, histogram: numbox…
svdhoog Jun 3, 2019
bc666f7
Updated parameters.py and plots.py: Time series, scatter plots: added…
svdhoog Jun 3, 2019
69664da
Updated main.py: one memory optimization
svdhoog Jun 3, 2019
d585600
Updates to FLAViz plots.py; Unified plot_names to: AgentName_Variable…
svdhoog Jun 4, 2019
9d83f77
Fixed error in db_hdf5_v2 if output folder (h5) does not exist
svdhoog Jun 7, 2019
b007a5d
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Jun 7, 2019
688d340
Issue #9 Fixed error in db_hdf5_v2 if output folder (h5) does not exist
svdhoog Jun 7, 2019
394a43d
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Jun 7, 2019
d2b9d13
Issue #14: Fix yaml loader
svdhoog Jun 9, 2019
54b7dae
Add test_data folder
svdhoog Jun 9, 2019
099da45
Minor edits to readme
svdhoog Jun 11, 2019
c133465
Added readme for test data circles model
svdhoog Jun 11, 2019
575bd6f
Added readme for test data circles model
svdhoog Jun 11, 2019
c79f1c1
Minor edits to readme
svdhoog Jun 11, 2019
e5f01c7
Edits to readme circle model
svdhoog Jun 11, 2019
649a73a
Add data for circles model Dataset 1
svdhoog Jun 11, 2019
fad7c9a
Edit config files for circles model datasets
svdhoog Jun 11, 2019
d2638ce
Edit readme for circles model
svdhoog Jun 11, 2019
8a61ce2
Edit readme for circles model
svdhoog Jun 11, 2019
4f722cf
Edit readme for circles model
svdhoog Jun 11, 2019
e127c44
Edit readme for circles model
svdhoog Jun 11, 2019
2854a33
Edit readme for circles model
svdhoog Jun 11, 2019
9356975
Edit readme for circles model
svdhoog Jun 11, 2019
2722644
Edit readme for circles model
svdhoog Jun 11, 2019
3b93728
Edit readme for circles model
svdhoog Jun 11, 2019
c785dbf
Edit readme for circles model
svdhoog Jun 11, 2019
bd3b080
Edit main readme
svdhoog Jun 11, 2019
65990d7
Edit readme for circles model
svdhoog Jun 11, 2019
c4a2891
Issue #33: Scatterplots, added case many plot, summary=full
svdhoog Jun 13, 2019
19f9725
Issue #20: Test cases for Scatterplots
svdhoog Jun 13, 2019
2044f9b
Merge pull request #34 from svdhoog/dev_scatterplots
svdhoog Jun 13, 2019
4bcdf1f
Issue #20: Test cases for Scatterplots plot60
svdhoog Jun 13, 2019
e806264
Merge pull request #36 from svdhoog/dev_scatterplots
svdhoog Jun 13, 2019
0e3215d
Update readme.rst
svdhoog Jun 13, 2019
3d80df5
Issue #20: Test cases for Scatterplots; final fixes filenames
svdhoog Jun 14, 2019
bb5d359
Merge pull request #37 from svdhoog/dev_scatterplots
svdhoog Jun 14, 2019
d0c5602
Test circles model: FLAME code update
svdhoog Jun 14, 2019
475e73d
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Jun 14, 2019
b4297dc
Issue #20: Timeseries - added new plots
svdhoog Jun 14, 2019
fec599a
FLAViz src files: updates Timeseries plots
svdhoog Jun 14, 2019
cee03a0
FLAViz Scatterplots Timeseries - example plots
svdhoog Jun 14, 2019
761febd
Merge pull request #38 from svdhoog/dev_timeseries
svdhoog Jun 14, 2019
d347b66
Issue #15 Timeseries plot10
svdhoog Jun 16, 2019
cc5f5a8
Minor edits
svdhoog Apr 12, 2021
f622fb9
Issue #39: Fixed Panel HDF5 issue; now using DataFrame to store
svdhoog Apr 13, 2021
df3e431
Issue #39: Test data for db->h5 conversion
svdhoog Apr 13, 2021
5446c8e
Issue #28: Fixed description of merge_hdf_agentwise
svdhoog Apr 13, 2021
833a625
Update README.md
svdhoog Apr 13, 2021
3abaf20
Merge pull request #40 from svdhoog/dev-db-conversion-dataframes
svdhoog Apr 13, 2021
645adf1
Issue #41: Fixes col name in df from __ITERATIONS to iter
svdhoog Apr 14, 2021
1336f2f
Merge pull request #42 from svdhoog/dev-db-conversion-dataframes
svdhoog Apr 14, 2021
ba2731b
Issue #43: Adding test scripts
svdhoog Apr 14, 2021
1bd74f7
Merge branch 'master' of https://github.com/svdhoog/ETACE
svdhoog Apr 14, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
452 changes: 452 additions & 0 deletions .gitignore

Large diffs are not rendered by default.

4 changes: 4 additions & 0 deletions AUTHORS
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Authors of FLAViz:

* Krishna Raj Devkota <kdevkota@gmail.com> (ETACE, Bielefeld University)
* Sander van der Hoog <svdhoog@gmail.com> (ETACE, Bielefeld University)
674 changes: 674 additions & 0 deletions LICENSE

Large diffs are not rendered by default.

144 changes: 133 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,137 @@
# Data processing and analyzing scripts
FLAViz: Flexible Large-scale Agent Visualization Library

#### Depends
-------------------------------------------------------------------------------

Python core libraries
Pandas
matplotlib
h5py
pytables
scipi
numy
Copyright (c) 2018 Sander van der Hoog
If you have any problems or enquiries, you can contact the
library maintainer at <svdhoog@gmail.com>

#### Usage
-------------------------------------------------------------------------------

Refer to individual folders
The Flexible Large-scale Agent Visualization Library (FLAViz) is a data analysis and agent visualization library developed for agent-based simulation models created using the FLAME simulation environment.

Agent-based simulations typically generate data spanning multiple dimensions, e.g. parameter sets, Monte Carlo replication runs, different agent types, many agent instances per type, many variables per agent, and the time dimension.

To deal with such a large heterogeneity and variety of data, it should be stored as an organized, high-dimensional data set, to allow for proper data aggregation, filtering, selection, slicing etc.

FLAViz builds on Python pandas and matplotlib, and can be used to filter, transform, and visualize time series data that is stored using hierarchical levels in the HDF5 file format. Various plotting styles can be specified, e.g., time series, box plots, scatter plots, histograms, and delay- or phase plots.

-------------------------------------------------------------------------------
Features
--
v1.0 - First release

* data conversion from XML to SQLite DB and HDF5
* data module using Python pandas
* visualization module based on matplotlib


Getting started
--

For more information on how to use the library, view the [manual](https://github.com/svdhoog/FLAViz/blob/master/docs/manual/tex/FLAViz_Manual.pdf).

Many example plots are [here](https://github.com/svdhoog/FLAViz/tree/master/docs/visualisation_scripts/readme.rst).

To get started immediately, you can run the library out-of-the-box using the [test data](https://github.com/svdhoog/FLAViz/tree/master/data/visualisation). Instructions on how to do this can be found in the section on Testing in the [tutorial](https://github.com/svdhoog/FLAViz/tree/master/docs/visualisation_scripts/readme.rst).

Test data
--

A simple model with test data is provided in the folder [circles model](https://github.com/svdhoog/FLAViz/tree/master/data/visualisation/models/circle). A description is in the [readme file](https://github.com/svdhoog/FLAViz/tree/master/data/visualisation/models/circle/readme.rst).


Requirements:
--

You will need to satisfy the following dependencies.

For the FLAViz visualization scripts:

* python3 (>= 3.4) Python3 core libraries
* python3-h5py (>= 2.2.1) -- needed for HDF5 files to be re-writable, appendable
* python3-matplotlib (>= 2.2.2)
* python3-numexp (>= 2.6)
* python3-numpy (>= 1.14.3)
* python3-pandas (0.22.0 or >= 0.19.0)
* python3-pyTables (>= 3.4) -- currently used for HDF5 files, using the 'tables' format (slower than 'fixed' format, but allows for appendable HDFStores)
* python3-scipy (1.1.0)
* python3-tk (>= 3.4.3)
* python3-yaml (>= 3.12)
* python3-lxml (4.2.1 or >= 3.3.3)

For the data conversion scripts:

* sqlite3 (>= 3.8.2)
* python3-pandas (0.22.0 or >= 0.19.0)
* python3-lxml (4.2.1 or >= 3.3.3)

Installation:
--

To see which versions of the Python3 packages you have installed, run:

```
$ ls /usr/local/lib/python3.4/dist-packages -lh
```

Notes:
--
On some platforms the package manager will install older versions of the above packages.
For example, in Linux Mint 17 (Quiana), the version of matplotlib that is installed using apt-get is 1.19 instead of the newer version 2.2.2 (which is required).

The following packages can be installed via your distribution's package manager:

```
$ sudo apt-get install sqlite3 python3-pip python3-lxml python3-yaml python3-numexp python3-tables python3-tk
```

To install the latest versions of the Python numerical packages, you can use the python3-pip command:

```
$ python3 -mpip install -U matplotlib numpy pandas scipy
```

Alternatively, you can us pip3 directly:

```
$ pip3 install matplotlib numpy pandas scipy
```

Running the main module
--

The main Python script has a single argument, which is the path to the folder containing configuration files:
To start the library, you should run it from the root FLAViz folder:

```
$ python3 /src/visualisation_scripts/main.py -p /src/visualisation_scripts/config
```
where `config` is the path to the config folder containing the configuration files:

* config.yaml
* plot_config.yaml
* config_transform.yaml [optional]

For more information please see the `docs/` folder.


Contributors
------------
- [Sander van der Hoog](https://github.com/svdhoog)
- [Krishna Raj Devkota](https://github.com/krdevkota)
- [Fabian Herrmann](https://github.com/0xfabi)

Contributing
------------

1. Fork it
2. Create your feature branch (`git checkout -b my-new-feature`)
3. Commit your changes (`git commit -am 'Added some feature'`)
4. Push to the branch (`git push origin my-new-feature`)
5. Create new Pull Request

Copyright
---------

Copyright (c) 2018 Sander van der Hoog. See LICENSE for further details.
47 changes: 0 additions & 47 deletions TODO/alternate.py

This file was deleted.

30 changes: 0 additions & 30 deletions TODO/doc/dataset.rst

This file was deleted.

Loading