RESTful web-services in JSIDPlay2

The emulator can be started in server mode and gives access to various functionalities like real-time streaming of SIDs or audio recognition and more. You can implement your own client to access this emulator. The server mode of JSIDPlay2 is called "JSIDPlay2Server"

The service methods are available running the server as a command-line tool explained in this document or as an alternative by clicking start in JSIDPlay2 user interface version.

Info: All commands showed here have been verified on Ubuntu 18.

Info: If a script in this document has a backslash at the end of a line, this is for better readability and must not be entered for real.

Preparation

General

All the following RESTful service methods of JSIDPlay2Server are using your configured music collections HVSC and CGSC.

To make the following examples to work, first configure HVSC and CGSC music collection in JSIDPlay2, properly!

If you can browse these music collections in JSIDPlay2 the service methods can browse and stream the contained tune files as well.

Security

Basic authentication is used calling the services with the following pre-configured credentials:

Username: jsidplay2
Password: jsidplay2!

You can change the password of the user or add an administrative user by an external configuration file.

The file must be named: "tomcat-users.xml" located in your home directory and be readable on startup of the server.

The reason why you can add an administrative user is to grant access to additional private music collections which normal users are not able to access.

Example:

<tomcat-users>
  <role rolename="admin"/>
  <role rolename="user"/>

  <user name="jsidplay2" password="jsidplay2!" roles="user"/>
  <user name="admin" password="admin" roles="admin"/>
</tomcat-users>

All known roles are user and admin.

JSIDPlay2Server supports HTTP and/or HTTPS protocol for client to server communication.

Use Self-signed certificate

Info: This step is only required, if you want to make use of the HTTPS protocol.

To create a self-signed certificate you must use the keytool command of the java installation.

keytool -genkey -alias jsidplay2 -keyalg RSA -keystore /etc/jsidplay2.ks -keysize 2048 (1)

Warning: A self-signed certificate is only a temporary solution, since it creates a warning message in the browser window and is unaccepted by iPhones and mobiles running Android9 and later versions!

Use trusted certificate

Info: This step is only required, if you want to make use of the HTTPS protocol.

Info: This example is related to Let’s Encrypt and Apache2 installation on Linux according to my situation here.

If you as well, already have a web-server and want to host JSIDPlay2-server on the same machine, then you can use HTTPS using the same certificate for both: your web-server and JSIDPlay2Server. This is a step by step instruction of how it works:

First lets make the Apache2 web-server secure, which is very easy with Let’s Encrypt, these days.

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-apache
sudo certbot --apache

Valid certificates do not last forever, therefore for automatic renewal a cron job is created by Let’s Encrypt. Let’s Encrypt already adds the certbot script to renew the certificates regularly.

Important: In order to re-use these renewed certificates for JSIDPlay2 we add a cronjob as well (script /usr/bin/jsidplay2cert). Our cronjob creates the Java keystore (/etc/jsidplay2.ks) out of the certificates renewed by Let’s Encrypt.

To configure that cronjob type 'sudo crontab -e' and add the last line containing jsidplay2cert.

#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
40 3 * * 0 certbot -q renew
@weekly root jsidplay2cert # (1)

jsidplay2cert is a script we have to execute in order to create our keystore (/etc/jsidplay2.ks), weekly

Now create the script to be executed by cronjob typing 'sudo vi /usr/bin/jsidplay2cert'.

#!/bin/bash

# create one big certificate file PEM
cat /etc/letsencrypt/live/haendel.ddns.net/{cert.pem,chain.pem,fullchain.pem,privkey.pem} \
	> /tmp/fullchain.pem

# convert certificate file PEM to PKCS12 required by keytool
openssl pkcs12 -export -out /tmp/fullchain.pkcs12 -in /tmp/fullchain.pem \
	-password pass:jsidplay2 >> /var/log/jsidplay2cert.log 2>&1
rm /tmp/fullchain.pem >> /var/log/jsidplay2cert.log 2>&1

# Create empty keystore, first we create a non-empty one, then we remove contents again
rm /etc/jsidplay2.ks >> /var/log/jsidplay2cert.log 2>&1
keytool -genkey -keyalg RSA -alias jsidplay2 -keystore /etc/jsidplay2.ks \
	-storepass jsidplay2 -keypass jsidplay2 \
	-dname 'cn="JSIDPlay2", ou="JSIDPlay2", o="Open Source", l="Berlin", c=DE' \
	>> /var/log/jsidplay2cert.log 2>&1
/usr/bin/keytool -delete -alias jsidplay2 -keystore /etc/jsidplay2.ks -storepass jsidplay2 \
	>> /var/log/jsidplay2cert.log 2>&1

# Import certificate into keystore, first we import the certificates, \
	then we need to change the alias name
keytool -v -importkeystore -srckeystore /tmp/fullchain.pkcs12 \
	-destkeystore /etc/jsidplay2.ks -deststoretype JKS -srcstorepass jsidplay2 \
	-deststorepass jsidplay2 >> /var/log/jsidplay2cert.log 2>&1
rm /tmp/fullchain.pkcs12 >> /var/log/jsidplay2cert.log 2>&1
keytool -keystore /etc/jsidplay2.ks -changealias -alias 1 -destalias jsidplay2 \
	-storepass jsidplay2 >> /var/log/jsidplay2cert.log 2>&1

sudo -i -u ken /home/ken/jsidplay2-server.sh >> /var/log/jsidplay2cert.log 2>&1 # (1)

At last our server gets restarted with the renewed certificate, please refer to Launch JSIDPlay2Server using HTTPS

Now, grant permission to execute that script by our cronjob:

sudo chmod 755 /usr/bin/jsidplay2cert

As a result we get weekly a fresh new keystore (/etc/jsidplay2.ks). A log file for troubleshooting is placed here (/var/log/jsidplay2cert.log)

Launch JSIDPlay2Server using HTTP

First lets explain how to start JSIDPlay2Server in general for unencrypted HTTP connections and without the created keystore above.

Info: You can start JSIDPlay2Server standalone using the following command in a console window, instead of starting the UI version of JSIDPlay2.

To start JSIPlay2Server without HTTPS, but only HTTP, please use the following command.

java -classpath jsidplay2-4.10-ui.jar server.restful.JSIDPlay2Server # (1)

Launch the JSIDPlay2Server standalone

For all supported parameters, please type:

java -classpath jsidplay2-4.10-ui.jar server.restful.JSIDPlay2Server --help # (1)

Show usage of the JSIDPlay2Server standalone

Launch JSIDPlay2Server using HTTPS

Now lets explain how to start JSIDPlay2Server using encrypted HTTPS connections with the keystore created above.

To start the JSIDPlay2Server with HTTPS using the formerly created keystore, you can use the following script, please type 'vi ~/jsidplay2-server.sh'.

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.10-linux
pkill -f server.restful.JSIDPlay2Server
java -classpath jsidplay2-4.10-ui.jar server.restful.JSIDPlay2Server \
	--appServerKeystore /home/ken/jsidplay2.ks \
	--appServerKeystorePassword jsidplay2 \
	--appServerKeyAlias jsidplay2 \
	--appServerKeyPassword jsidplay2 \
	--appServerConnectors HTTPS & # (1)

Launch the JSIDPlay2Server using HTTPS standalone

Note: Parameter appServerConnectors controls if we want to support HTTP, HTTPS or both!

Warning: The passwords will always be deleted after exit of JSIDPlay2 to not appear in the configuration file of JSIDPlay2 for security reasons!

Now, grant permission to execute that script:

sudo chmod 755 ~/jsidplay2-server.sh

Now, we are finished to launch JSIDPlay2 using HTTP or HTTPS

Create database for WhatsSID? tune recognition

JSIDPlay2Server can optionally serve audio recognition requests to answer the question: What tune is currently played. This is similar to what the commercial Shazam app does.

As a preparation to handle these requests you must create a database beforehand. You can choose your own database, but in these examples I will use mysql.

First install and configure a database and create a user with a desired password.

sudo apt-get install mysql-server
sudo mysql_secure_installation utility

sudo mysql -u root -p
CREATE USER '<username>'@'localhost' IDENTIFIED BY '<password>'; # (1)
GRANT ALL PRIVILEGES ON *.* TO '<username>'@'localhost' IDENTIFIED BY '<password>'; # (2)
quit

Replace <username> and <password> according to your own choice
grant priviledges to the newly created user

Allow remote connections:

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
bind-address            = 0.0.0.0

Prepare database configuration:

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf

max_allowed_packet      = 128M # (1)
innodb_buffer_pool_size = 32G # (2)
sql-mode                = "ANSI_QUOTES" # (3)

increase packet size (because adding a song requires sending a larger package)
use 75% of your physical RAM for the database to keep it preferably entirely in memory.
use ANSI quotes for SQL statements in case you want to use client software like org.hsqldb.util.DatabaseManagerSwing to have a look into your database, because these use ANSI quoted sql.

Enable autostart on boot time and restart service:

sudo systemctl enable mysql # (1)
sudo systemctl restart mysql # (2)

launch mysql on startup
restart mysql service

Now start the WhatsSID database creator to create and insert the tune fingerprintings for your music collection. But be warned, this process will run very very long around 30 days on my machine for HVSC, because this will create WAV files of the entire collection (8KHz mono recordings) before adding the fingerprint. However, it is safe to stop and restart the process if required, because it checks if WAV is already present and if database already contains the song.

To start the process you can use the following script, please type 'vi ~/fingerprint-creator.sh'.

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.10-linux
pkill -f server.restful.JSIDPlay2Server
java -Dhibernate.dialect.storage_engine=innodb \
	-classpath jsidplay2-4.10-ui.jar ui.tools.RecordingTool \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc80?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL55Dialect \
	--fingerprinting true \
	--hvsc <pathToHVSC> \ # (1)
	<pathToMusicCollection> & # (2)

pathToHVSC is used to read song length file database contained in HVSC
pathToMusicCollection is the base directory to recursively create audio fingerprintings. This can be pathToHVSC or the path for CGSC!

Note: The pathToMusicCollection is traversed recursively and every song is converted to a 8KHz WAV recording. You will need about 250GB disk space to store them! After every created WAV file an audio fingerprint is created and stored in the database.

Now, grant permission to execute that script:

sudo chmod 755 ~/fingerprint-creator.sh

Now you can use WhatsSID? tune recognition.

Update of WhatsSID database

Updates of WhatsSID database are done in much shorter times. Last update to hvsc80 took around 24 hours. A new database will be created (in the following example named "hvsc80"). It re-uses WAV files of unmodified tunes compared to the last HVSC version. Therefore you need to specify the previously used HVSC directory containing all the WAV files which are so time consuming to create from scratch. The newly created HVSC folder will get copied recordings from there and if tunes were changed or added, only they will be created from scratch.

Keep in mind in the end the update requires another 250GB of disk space for the new WAV recordings.

Please change the script fingerprint-creator.sh to add the previousDirectory parameter:

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.10-linux
pkill -f server.restful.JSIDPlay2Server
java -Dhibernate.dialect.storage_engine=innodb \
	-classpath jsidplay2-4.10-ui.jar ui.tools.RecordingTool \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc80?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL55Dialect \
	--fingerprinting true \
	--hvsc <pathToHVSC> \ # (1)
	--previousDirectory <previousPathToHVSC> \ # (2)
	<pathToMusicCollection> & # (3)

pathToHVSC is used to read song length file database contained in HVSC
previousPathToHVSC is the formerly used pathToHVSC containing WAV recordings among the SID files.
pathToMusicCollection is the base directory to recursively create audio fingerprintings. This can be pathToHVSC or the path for CGSC!

Launch JSIDPlay2Server with additional WhatsSID? tune recognition

Now lets explain how to start JSIDPlay2Server providing audio recognition.

Note: You must create a database beforehand, that was explained in the previous section.

You only have to add additional WhatsSID database parameters.

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.10-linux
pkill -f server.restful.JSIDPlay2Server
java -Dhibernate.dialect.storage_engine=innodb \
	-classpath jsidplay2-4.10-ui.jar server.restful.JSIDPlay2Server \
	--appServerKeystore /home/ken/jsidplay2.ks \
	--appServerKeystorePassword jsidplay2 \
	--appServerKeyAlias jsidplay2 \
	--appServerKeyPassword jsidplay2 \
	--appServerConnectors HTTPS \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc80?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&autoReconnect=true" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL55Dialect & # (1)

Launch the JSIDPlay2Server using WhatsSID? tune recognition

Autostart JSIDPlay2Server on system startup

To start JSIPlay2Server on system startup, please type 'vi /etc/systemd/system/jsidplay2.service'.

[Unit]
Wants=network-online.target
After=network.target network-online.target

[Service]
ExecStart=java \
	-Dhibernate.dialect.storage_engine=innodb \
	-classpath /home/ken/Downloads/jsidplay2-4.10-linux/jsidplay2-4.10-ui.jar \
	server.restful.JSIDPlay2Server \
	--appServerKeystore /home/ken/jsidplay2.ks \
	--appServerKeystorePassword jsidplay2 \
	--appServerKeyAlias jsidplay2 \
	--appServerKeyPassword jsidplay2 \
	--appServerConnectors HTTP_HTTPS \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc80?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL5InnoDBDialect
User=ken

[Install]
WantedBy=default.target

Now, grant permission and enable jsidplay2.service to autostart:

sudo chmod u+x /etc/systemd/system/jsidplay2.service # (1)
sudo systemctl start jsidplay2.service # (2)
sudo systemctl enable jsidplay2.service # (3)
sudo systemctl stop jsidplay2.service # (4)

Make script executable
Optional: start service for testing
Enable service to run at boot
Optional: stop service after testing

Access from the Internet

To use your client from within your private local area network does not require additional preparations, you just need to know and reach the IP address, where JSIDPlay2 is running on.

But, if you want to use a client from any location in the internet, that would require some additional preparations:

You will need a hostname which resolves to the IP-address of your always reachable server, where JSIDPlay2 is running on (e.g. using a provider like https://freeddns.noip.com and configure dynamic DNS inside your router)
You will need to configure your router to redirect requests to that server: I mean port forwarding. You should forward requests using the port, that you configured for the JSIDPlay2Server (8080 and 8443). This will make it necessary to configure a fixed IP address for your server within your local area network, beforehand. Now you can forward all related traffic to your server where JSIDPlay2 is running on.

Warning: Keep in mind, that opening ports in your firewall will raise the security risk. You will make yourself vulnerable to attacks from hackers.

Note: I will not take responsability for any risks or damages. Do this on your own risk!

JSIDPlay2Server - RESTful service endpoints

Info: Depending on the connection type of JSIDPlay2Server you have to use HTTP or HTTPS as protocol and port 8080 (HTTP) or 8443 (HTTPS). Please refer the command-line parameters appServerConnectors, appServerPort and appServerSecurePort!

Get all SID filter names (required to stream SID as MP3 later)

https://haendel.ddns.net:8443/jsidplay2service/JSIDPlay2REST/filters

Note: SID filter names are prefixed with the emulation (RESID or RESIDFP) and the SID model (MOS6581 or MOS8580) and appended by their name, e.g. RESID_MOS8580_FilterAverage8580, That way filters can be grouped or sorted on the client side.