In September 2017 Java 9 was released and changed many things. First thing you have to do: Update to the JTS3ServerMod 6.4.2 or newer, it don't need the add-modules part anymore!
Since JTS3ServerMod 6.5.2 there are no compatibility issues and warning messages anymore, should be fully compatible with future Java versions.