Database to store all data from crypto exchanges, currently working with Binance, Bittrex, Cryptopia and Poloniex.
Can be used for technical analysis, bots, backtest, realtime trading, etc.
This install guide was made for Ubuntu 16.04+. Will need some adjustments to work with other distros.
Docker directory has a default configuration that allows users to implement an pre configured database, ready to receive data from Exchanges and use it. In order to use Influxdb Docker container is only necessary to follow the steps bellow.
git clone https://github.com/ivopetiz/crypto-database.git
cd crypto-database
. docker/build
. docker/start
And your Influxdb crypto database throw Docker container should be ready to be used.
Start by installing Golang, to build the applications responsible for populate Crypto-database.
sudo apt-get install golang
After install Golang, you will need to install InfluxDB. Chronograf is also recomended.
sudo apt-get install influxdb influxdb-client chronograf
You can also get InfluxDB last versions from Influx website. This version of Crypto-database where tested with Influxdb 1.5.3 and Chronograf 1.5.0. You can install these packages by running:
wget https://dl.influxdata.com/influxdb/releases/influxdb_1.5.3_amd64.deb
sudo dpkg -i influxdb_1.5.3_amd64.deb
wget https://dl.influxdata.com/chronograf/releases/chronograf_1.5.0.0_amd64.deb
sudo dpkg -i chronograf_1.5.0.0_amd64.deb
Start Influxdb and then run Influxdb prompt.
sudo systemctl enable influxdb.service
sudo systemctl start influxdb.service
influx
Create databases and user with privileges to use databases:
CREATE DATABASE altcoin
CREATE DATABASE balance
USE altcoin
CREATE USER <username> WITH PASSWORD '<password>' WITH ALL PRIVILEGES
GRANT ALL PRIVILEGES TO <username>
Before build crypto markets and balance applications is necessary to clone this repository.
git clone https://github.com/ivopetiz/crypto-database.git
After cloning this rep, is necessary to configure Influxdb user, password and server on system. One option is to define variables on system in order to hide it from git. Variables can also be defined on consts.go.
DBUSER=<your-db-user>
DBPASS=<your-db-password>
SERVERDB=<your-db-server>
By default, data is recorded every 10 seconds but this value can be changed. Timeout can be defined on consts.go. A timeout too big won't present fast changes on prices. By the other hand, a timeout too small will make your IP address blocked on crypto exchanges, that only allows a certain number of request per minute. All exchanges have different limits and you can consult these values on exchanges API official websites.
Log file will be stored on /log/altdb_coin.log. This path can be changed on main.go. Make sure you have the right privileges to write in /log/.
sudo mkdir /log
sudo touch /log/altdb_coin.log
sudo chown <user>:<user> /log/altdb_coin.log
sudo chmod 644 /log/altdb_coin.log
Compile market data getter executable.
cd markets
go get
go build -o markets -ldflags="-s -w" main.go consts.go
Go lang will return an executable file called markets. Now you need to run in order to populate database. One option is to use markets as a service, in order to keep tracking of it. To run markets as a service is necessary to create /etc/systemd/system/cryptomarket.service with the following content:
[Unit]
Description=Service running a crypto market data getter.
After=network.target
[Service]
Type=simple
User=user
WorkingDirectory=/home/user/crypto-database
ExecStart=/home/user/crypto-database/markets
Restart=always
[Install]
WantedBy=multi-user.target
After save cryptomarket.service file, run the commands bellow:
sudo systemctl daemon-reload
sudo systemctl enable cryptomarket.service
sudo systemctl start cryptomarket.service
Now markets will run as a service, starting when OS initializes and recovers in case of failure.
If you want to add your balance to DB, you will need to generate your API key on crypto exchanges, in order to validate your login and get your data. Currently working only with Bittrex exchange.
To generate an API key and secret, you can access crypto exchanges website, on definitions part. Make sure you keep your API key and API secret secret and give this key minimum permissions, in order to obtain just balance info. With bad permissions, any person with this key and secret can consult, trade or withdraw your coins.
After get API key and secret from exchanges, you need to add it to balance.go. One option is to define system variables:
BITTREX_API_KEY=<your-bittrex-api-key>
BITTREX_API_SECRET=<your-bittrex-api-secret>
After define API_KEY and API_SECRET in balance.go is necessary to build balance program:
cd balance
go get
go build -o balance -ldflags="-s -w" balance.go
Once balance is not supposed to run every minute it wasn't built with a timeout, so user need to run it everytime. This can be done automatically with a Crontab rule or as a service, like was done with markets.service.
To add new balance info to DB every hour, add a crontab rule by running crontab -e and add the following line:
0 * * * * /home/user/crypto-database/balance
The above rule will run balance every hour at :00.
Chronograf presents crypto data from Influxdb. Can be particularly useful to plot data or to quick check market prices. Chronograf is easy to use and it's only necessary to configure with your Influxdb definitions.
In order to start service, after installation will need to run:
sudo systemctl start chronograf.service
If you use Influxdb in the same machine as Chronograf service and have used the default configs, Influxdb will be in https://localhost:8086.
You will also need to input your Influxdb user and password in order to give Chronograf access to DB.
If Influxdb and Chronograf services are in diferent machines, you will need to change localhost by Influxdb machine IP address.
Chronograf service uses port 8888. Port can be changed on /etc/default/chronograf.
Anyone can access Chronograf but is possible to block access from other machines. In config file you can change all configs.
By default, Chronograf config file will be looking like this:
HOST=0.0.0.0
PORT=8888
HOST will define who can access Chronograf website. It can be blocked to localhost machine or other specific IP address which can be a good option in terms of security.
- add more exchanges to Balance
- makefile