RESTful web-services in JSIDPlay2

The emulator can be started in server mode and gives access to various functionalities like real-time streaming of SIDs or audio recognition and more. You can implement your own client to access this emulator. The server mode of JSIDPlay2 is called "JSIDPlay2 AppServer"

The service methods are available running the server as a command-line tool explained in this document or as an alternative by clicking start in JSIDPlay2 user interface version.

Info: All commands showed here have been verified on Ubuntu 18.

Info: If a script in this document has a backslash at the end of a line, this is for better readability and must not be entered for real.

Preparation

General

All the following RESTful service methods of JSIDPlay2 AppServer are using your configured music collections HVSC and CGSC.

To make the following examples to work, first configure HVSC and CGSC music collection in JSIDPlay2, properly!

If you can browse these music collections in JSIDPlay2 the service methods can browse and stream the contained tune files as well.

Security

Basic authentication is used calling the services with the following pre-configured credentials:

  • Username: jsidplay2

  • Password: jsidplay2!

You can change the password of the user or add an administrative user by an external configuration file.

The file must be named: "tomcat-users.xml" located in your home directory and be readable on startup of the server.

The reason why you can add an administrative user is to grant access to additional private music collections which normal users are not able to access.

Example:

<tomcat-users>
  <role rolename="admin"/>
  <role rolename="user"/>

  <user name="jsidplay2" password="jsidplay2!" roles="user"/>
  <user name="admin" password="admin" roles="admin"/>
</tomcat-users>

All known roles are user and admin.

The built-in AppServer in JSIDPlay2 supports HTTP and/or HTTPS protocol for client to server communication.

Use Self-signed certificate

Info: This step is only required, if you want to make use of the HTTPS protocol.

To create a self-signed certificate you must use the keytool command of the java installation.

keytool -genkey -alias jsidplay2 -keyalg RSA -keystore /etc/jsidplay2.ks -keysize 2048 (1)
  1. Warning: A self-signed certificate is only a temporary solution, since it creates a warning message in the browser window and is unaccepted by iPhones and mobiles running Android9 and later versions!

Use trusted certificate

Info: This step is only required, if you want to make use of the HTTPS protocol.

Info: This example is related to Let’s Encrypt and Apache2 installation on Linux according to my situation here.

If you as well, already have a web-server and want to host JSIDPlay2-server on the same machine, then you can use HTTPS using the same certificate for both: your web-server and JSIDPlay2 AppServer. This is a step by step instruction of how it works:

First lets make the Apache2 web-server secure, which is very easy with Let’s Encrypt, these days.

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:certbot/certbot
sudo apt-get update
sudo apt-get install python-certbot-apache
sudo certbot --apache

Valid certificates do not last forever, therefore for automatic renewal a cron job is created by Let’s Encrypt. Let’s Encrypt already adds the certbot script to renew the certificates regularly.

Important: In order to re-use these renewed certificates for JSIDPlay2 we add a cronjob as well (script /usr/bin/jsidplay2cert). Our cronjob creates the Java keystore (/etc/jsidplay2.ks) out of the certificates renewed by Let’s Encrypt.

To configure that cronjob type 'sudo crontab -e' and add the last line containing jsidplay2cert.

#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h  dom mon dow   command
40 3 * * 0 certbot -q renew
@weekly root jsidplay2cert (1)
  1. jsidplay2cert is a script we have to execute in order to create our keystore (/etc/jsidplay2.ks), weekly

Now create the script to be executed by cronjob typing 'sudo vi /usr/bin/jsidplay2cert'.

#!/bin/bash

# create one big certificate file PEM
cat /etc/letsencrypt/live/haendel.ddns.net/{cert.pem,chain.pem,fullchain.pem,privkey.pem} \
	> /tmp/fullchain.pem

# convert certificate file PEM to PKCS12 required by keytool
openssl pkcs12 -export -out /tmp/fullchain.pkcs12 -in /tmp/fullchain.pem \
	-password pass:jsidplay2 >> /var/log/jsidplay2cert.log 2>&1
rm /tmp/fullchain.pem >> /var/log/jsidplay2cert.log 2>&1

# Create empty keystore, first we create a non-empty one, then we remove contents again
rm /etc/jsidplay2.ks >> /var/log/jsidplay2cert.log 2>&1
keytool -genkey -keyalg RSA -alias jsidplay2 -keystore /etc/jsidplay2.ks \
	-storepass jsidplay2 -keypass jsidplay2 \
	-dname 'cn="JSIDPlay2", ou="JSIDPlay2", o="Open Source", l="Berlin", c=DE' \
	>> /var/log/jsidplay2cert.log 2>&1
/usr/bin/keytool -delete -alias jsidplay2 -keystore /etc/jsidplay2.ks -storepass jsidplay2 \
	>> /var/log/jsidplay2cert.log 2>&1

# Import certificate into keystore, first we import the certificates, \
	then we need to change the alias name
keytool -v -importkeystore -srckeystore /tmp/fullchain.pkcs12 \
	-destkeystore /etc/jsidplay2.ks -deststoretype JKS -srcstorepass jsidplay2 \
	-deststorepass jsidplay2 >> /var/log/jsidplay2cert.log 2>&1
rm /tmp/fullchain.pkcs12 >> /var/log/jsidplay2cert.log 2>&1
keytool -keystore /etc/jsidplay2.ks -changealias -alias 1 -destalias jsidplay2 \
	-storepass jsidplay2 >> /var/log/jsidplay2cert.log 2>&1

sudo -i -u ken /home/ken/jsidplay2-server.sh >> /var/log/jsidplay2cert.log 2>&1 (1)
  1. At last our server gets restarted with the renewed certificate, please refer to Launch JSIDPlay2 AppServer using HTTPS

Now, grant permission to execute that script by our cronjob:

sudo chmod 755 /usr/bin/jsidplay2cert

As a result we get weekly a fresh new keystore (/etc/jsidplay2.ks). A log file for troubleshooting is placed here (/var/log/jsidplay2cert.log)

Launch JSIDPlay2 AppServer using HTTP

First lets explain how to start JSIDPlay2 AppServer in general for unencrypted HTTP connections and without the created keystore above.

Info: You can start the built-in AppServer standalone using the following command in a console window, instead of starting the UI version of JSIDPlay2.

To start the JSIPlay2 AppServer without HTTPS, but only HTTP, please use the following command.

java -classpath jsidplay2-4.6.jar server.restful.JSIDPlay2Server (1)
  1. Launch the JSIDPlay2 AppServer standalone

For all supported parameters, please type:

java -classpath jsidplay2-4.6.jar server.restful.JSIDPlay2Server --help (1)
  1. Show usage of the JSIDPlay2 AppServer standalone

Launch JSIDPlay2 AppServer using HTTPS

Now lets explain how to start JSIDPlay2 AppServer using encrypted HTTPS connections with the keystore created above.

To start the JSIPlay2 AppServer with HTTPS using the formerly created keystore, you can use the following script, please type 'vi ~/jsidplay2-server.sh'.

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.6
pkill -f server.restful.JSIDPlay2Server
java -classpath jsidplay2-4.6.jar server.restful.JSIDPlay2Server \
	--appServerKeystore /home/ken/jsidplay2.ks \
	--appServerKeystorePassword jsidplay2 \
	--appServerKeyAlias jsidplay2 \
	--appServerKeyPassword jsidplay2 \
	--appServerConnectors HTTPS & (1)
  1. Launch the JSIDPlay2 AppServer using HTTPS standalone

Note: Parameter appServerConnectors controls if we want to support HTTP, HTTPS or both!

Warning: The passwords will always be deleted after exit of JSIDPlay2 to not appear in the configuration file of JSIDPlay2 for security reasons!

Now, grant permission to execute that script:

sudo chmod 755 ~/jsidplay2-server.sh

Now, we are finished to launch JSIDPlay2 using HTTP or HTTPS

Create database for WhatsSID? tune recognition

JSIDPlay2 AppServer can optionally serve audio recognition requests to answer the question: What tune is currently played. This is similar to what the commercial Shazam app does.

As a preparation to handle these requests you must create a database beforehand. You can choose your own database, but in these examples I will use mysql.

First install and configure a database and create a user with a desired password.

sudo apt-get install mysql-server
sudo mysql_secure_installation utility

sudo mysql -u root -p
CREATE USER '<username>'@'localhost' IDENTIFIED BY '<password>'; (1)
GRANT ALL PRIVILEGES ON *.* TO '<username>'@'localhost' IDENTIFIED BY '<password>'; (2)
quit
  1. Replace <username> and <password> according to your own choice

  2. grant priviledges to the newly created user

Allow remote connections:

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf
bind-address            = 0.0.0.0

Prepare database configuration:

sudo vi /etc/mysql/mysql.conf.d/mysqld.cnf

max_allowed_packet      = 64M (1)
innodb_buffer_pool_size = 16G (2)
sql-mode                = "ANSI_QUOTES" (3)
  1. increase packet size (because adding a song requires sending a larger package)

  2. use 75% of your physical RAM for the database to keep it preferably entirely in memory.

  3. use ANSI quotes for SQL statements in case you want to use client software like org.hsqldb.util.DatabaseManagerSwing to have a look into your database, because these use ANSI quoted sql.

Enable autostart on boot time and restart service:

sudo systemctl enable mysql (1)
sudo systemctl restart mysql (2)
  1. launch mysql on startup

  2. restart mysql service

Now start the WhatsSID database creator to create and insert the tune fingerprintings for your music collection. But be warned, this process will run very very long around 30 days on my machine for HVSC, because this will create WAV files of the entire collection (8KHz mono recordings) before adding the fingerprint. However, it is safe to stop and restart the process if required, because it checks if WAV is already present and if database already contains the song.

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.6
pkill -f server.restful.JSIDPlay2Server
java -Dhibernate.dialect.storage_engine=innodb \
	-classpath jsidplay2-4.6.jar ui.tools.FingerPrintingCreator \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc77?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL55Dialect \
	--hvsc <pathToHVSC>(1)
	<pathToMusicCollection> & (2)
  1. pathToHVSC is used to read song length file database contained in HVSC

  2. pathToMusicCollection is the base directory to recursively create audio fingerprintings. This can be pathToHVSC or the path for CGSC!

Note: The pathToMusicCollection is traversed recursively and every song is converted to a 8KHz WAV recording. You will need about 250GB disk space to store them! After every created WAV file an audio fingerprint is created and stored in the database.

Now you can use WhatsSID? tune recognition.

Update of WhatsSID database

Updates of WhatsSID database are done in much shorter times. Last update to hvsc77 took around 24 hours. A new database will be created (in the following example named "hvsc77"). It re-uses WAV files of unmodified tunes compared to the last HVSC version. Therefore you need to specify the previously used HVSC directory containing all the WAV files which are so time consuming to create from scratch. The newly created HVSC folder will get copied recordings from there and if tunes were changed or added, only they will be created from scratch.

Keep in mind in the end the update requires another 250GB of disk space for the new WAV recordings.

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.6
pkill -f server.restful.JSIDPlay2Server
java -Dhibernate.dialect.storage_engine=innodb \
	-classpath jsidplay2-4.6.jar ui.tools.FingerPrintingCreator \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc77?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL55Dialect \
	--hvsc <pathToHVSC> (1)
	--previousDirectory <previousPathToHVSC> (2)
	<pathToMusicCollection> & (3)
  1. pathToHVSC is used to read song length file database contained in HVSC

  2. previousPathToHVSC is the formerly used pathToHVSC containing WAV recordings among the SID files.

  3. pathToMusicCollection is the base directory to recursively create audio fingerprintings. This can be pathToHVSC or the path for CGSC!

Launch JSIDPlay2 AppServer with additional WhatsSID? tune recognition

Now lets explain how to start JSIDPlay2 AppServer providing audio recognition.

Note: You must create a database beforehand, that was explained in the previous section.

You only have to add additional WhatsSID database parameters.

#!/bin/bash -x
cd /home/ken/Downloads/jsidplay2-4.6
pkill -f server.restful.JSIDPlay2Server
java -Dhibernate.dialect.storage_engine=innodb \
	-classpath jsidplay2-4.6 server.restful.JSIDPlay2Server \
	--appServerKeystore /home/ken/jsidplay2.ks \
	--appServerKeystorePassword jsidplay2 \
	--appServerKeyAlias jsidplay2 \
	--appServerKeyPassword jsidplay2 \
	--appServerConnectors HTTPS \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc77?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC&autoReconnect=true" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL55Dialect & (1)
  1. Launch the JSIDPlay2 AppServer using WhatsSID? tune recognition

Autostart JSIDPlay2 AppServer on system startup

To start JSIPlay2 AppServer on system startup, please type 'vi /etc/systemd/system/jsidplay2.service'.

[Unit]
Wants=network-online.target
After=network.target network-online.target

[Service]
ExecStart=java \
	-Dhibernate.dialect.storage_engine=innodb \
	-classpath /home/ken/Downloads/jsidplay2-4.6/jsidplay2-4.6.jar \
	server.restful.JSIDPlay2Server \
	--appServerKeystore /home/ken/jsidplay2.ks \
	--appServerKeystorePassword jsidplay2 \
	--appServerKeyAlias jsidplay2 \
	--appServerKeyPassword jsidplay2 \
	--appServerConnectors HTTP_HTTPS \
	--whatsSIDDatabaseDriver com.mysql.cj.jdbc.Driver \
	--whatsSIDDatabaseUrl "jdbc:mysql://127.0.0.1:3306/hvsc77?createDatabaseIfNotExist=true&useUnicode=yes&characterEncoding=UTF-8&useJDBCCompliantTimezoneShift=true&useLegacyDatetimeCode=false&serverTimezone=UTC" \
	--whatsSIDDatabaseUsername <username> \
	--whatsSIDDatabasePassword <password> \
	--whatsSIDDatabaseDialect org.hibernate.dialect.MySQL5InnoDBDialect
User=ken

[Install]
WantedBy=default.target

Now, grant permission and enable jsidplay2.service to autostart:

sudo chmod u+x /etc/systemd/system/jsidplay2.service (1)
sudo systemctl start jsidplay2.service (2)
sudo systemctl enable jsidplay2.service (3)
sudo systemctl stop jsidplay2.service (4)
  1. Make script executable

  2. Optional: start service for testing

  3. Enable service to run at boot

  4. Optional: stop service after testing

Access from the Internet

To use your client from within your private local area network does not require additional preparations, you just need to know and reach the IP address, where JSIDPlay2 is running on.

But, if you want to use a client from any location in the internet, that would require some additional preparations:

  1. You will need a hostname which resolves to the IP-address of your always reachable server, where JSIDPlay2 is running on (e.g. using a provider like https://freeddns.noip.com and configure dynamic DNS inside your router)

  2. You will need to configure your router to redirect requests to that server: I mean port forwarding. You should forward requests using the port, that you configured for the built-in AppServer in JSIDPlay2 (8080 and 8443). This will make it necessary to configure a fixed IP address for your server within your local area network, beforehand. Now you can forward all related traffic to your server where JSIDPlay2 is running on.

Warning: Keep in mind, that opening ports in your firewall will raise the security risk. You will make yourself vulnerable to attacks from hackers.

Note: I will not take responsability for any risks or damages. Do this on your own risk!

JSIDPlay2 AppServer Usage of the RESTful services

Info: Depending on the connection type of JSIDPlay2 AppServer you have to use HTTP or HTTPS as protocol and port 8080 (HTTP) or 8443 (HTTPS). Please refer the command-line parameters appServerConnectors, appServerPort and appServerSecurePort!

Get all SID filter names (required to stream SID as MP3 later)

Note: SID filter names are prefixed with the emulation engine (RESID or RESIDFP) and the SID model (MOS6581 or MOS8580) and appended by their name, e.g. RESID_MOS8580_FilterAverage8580, That way filters can be grouped or sorted on the client side.

Get music collection directory

Example:

You can access any sub-directory of your music collection to navigate to the tunes you want to play on the client side. HVSC music collection root path starts with "/C64Music/" and CGSC music collection starts with "/CGSC/". Please append any sub-directory behind that root path to get the desired directory contents. Directory type entries are appended by a slash, whereas file type entries like tunes ends with their file extensions. To each directory contents a parent folder entry will be added appended by "../". Following that directory entry, you can easily navigate back to the parent directory. Additionally you can specify a file extension filter using the parameter filter, e.g. ".*\\.(sid|dat|mus|str|p00|prg|d64|mp3|mp4)$"

You can add more collections by creating an external configuration file located in your home directory.

The file must be named: "directoryServlet.properties" and be readable on startup of the server.

Example:

/MP3=/media/nas1/mp3,true (1)
/Assembly64=/media/nas1/Ken/C64/C64 Assembly64,false
  1. Syntax is: "<localDirectoryNameInTheRequest>" = "<realDirectoryName>", "<adminRoleRequired>"

In the request above simply use the URL "/jsidplay2service/JSIDPlay2REST/directory/MP3" or "/jsidplay2service/JSIDPlay2REST/directory/Assembly64" to access your collections.

Note: Admin role can restrict access to collections to users with an admin role.

Get tune infos

Example:

Return a list of information of the specified tune file.

Get contents of the first SID favorites tab

Return a list of favorite tune files.

Note: If there is more than one favorites list, the first non-empty one is returned

Get composer photo

Example:

Request a photo of a well-known tune composer to be displayed.

Stream SID as MP3

Example:

Return a mp3 stream of the specified tune. On the server side the emulator is started and streams the sound output back to the client. All parameters are used to specify emulation settings that should be used. Especially the MP3 parameters control the quality and size of the returned mp3 stream (vbr, cbr and vbrQuality). Using these parameter gives you the control about mobile phone transfer data volume and especially the costs that arise, if you stream over the internet using your specific mobile phone provider contract (as nobody has an unlimited flat rate these days). It is recommended to use less data volume with less precision (lower quality) for connections over the internet and higher data volume with more precision (higher quality) inside your private local network, e.g. WLAN connection.

I have implemented an example android app as a client for the built-in AppServer of JSIDPlay2. My android app uses constant bitrate of 64K for the internet and variable bitrate and highest quality for my private WLAN. This is according to my recommendation above.

Info: The audio parameter let you choose WAV as an alternative, but beware of a much bigger file sizes and costs.

Stream Demo as Video

Info: This service method does only work, if directoryServlet had been configured beforehand to grant access to "/Assembly64"!

Note: I will not take responsability for any costs, that arise from streaming sound or video files from the internet!

Info: All Parameter names match exactly the command line parameter names of the console player in gnu style (prepended by --). For example defaultLength=180 sets the default song length.

Info: The audio parameter let you choose AVI as an alternative, but beware of a much bigger file size.

Upload WAV for tune recognition

Info: This service method does only work, if server has been started with additional WhatsSID database parameters.

Request:

Http-Method: POST
Content-Type: application/json
Accept: application/json
{
"wav": "<Base64encodedWAV>" (1)
}
  1. replace <Base64encodedWAV> with the WAV file contents encoded with Base64.

This is a POST request to upload a WAV recording (short audio recording max. 20 seconds should be enough).

The WAV file must meet the following requirement:

  • Sample size must be 16 bits

  • encoding must be signed (one short per sample -32768..32767)

  • expected endianess is little endian

You are flexible to send:

  • mono or stereo

  • sampling frequencies 8KHz, 44.1KHz, 48KHz or 96KHz (but please use 8KHz to reduce size of the request)

Example response:

{
  "musicInfo": {
    "title": "Lightspeed",
    "artist": "Ari Yliaho (Agemixer)",
    "album": "2001 Scallop",
    "fileDir": "/media/nas2/Ken/C64/C64Music/MUSICIANS/A/Agemixer/Lightspeed.sid",
    "infoDir": "/C64Music/MUSICIANS/A/Agemixer/Lightspeed.sid",
    "audioLength": 332.0562438964844
  },
  "relativeConfidence": 18.2648401826484,
  "offsetSeconds": 72.12903225806451,
  "confidence": 80,
  "offset": 2236
}

MusicInfo consists of HVSC entries title, author and release mapped to title, artist and album fields. InfoDir is the path within the HVSC.

As an alternative you can send XML as well:

Request:

Http-Method: POST
Content-Type: application/xml
Accept: application/xml
<wav><wav><Base64encodedWAV></wav></wav> (1)
  1. replace <Base64encodedWAV> with the WAV file contents encoded with Base64

Example response:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<musicInfoWithConfidence>
  <musicInfo>
    <title>Lightspeed</title>
    <artist>Ari Yliaho (Agemixer)</artist>
    <album>2001 Scallop</album>
<fileDir>/media/nas2/Ken/C64/C64Music/MUSICIANS/A/Agemixer/Lightspeed.sid</fileDir>
<infoDir>/C64Music/MUSICIANS/A/Agemixer/Lightspeed.sid</infoDir>
    <audioLength>332.0562438964844</audioLength>
  </musicInfo>
  <confidence>80</confidence>
<relativeConfidence>18.2648401826484</relativeConfidence>
  <offsetSeconds>72.12903225806451</offsetSeconds>
  <offset>2236</offset>
</musicInfoWithConfidence>

As a second alternative you can upload the WAV file as a mime/multipart like a file upload in a browser does. Only the first file gets analyzed, though.

Note: The confidence level in the response gives you an idea of how certain the tune has been identified. The bigger the better.

Example Clients for JSIDPlay2 AppServer

Source code of the example Android app using the RESTful web-service interface can be found here.

Click here to get the Installer for JSIDPlay2 App.

Please note: Streaming music using your mobile can cause additional costs!

As an alternative you can use a browser enabled client like my example VUE client here