I was browsing in the file manager and noticed that I have a public_html and a www directory, both containing the same files/folders.
I changed the name of the plugin directory in the www directory and this seemed to work because the admin UI showed that I didn't have any plugins.
Was I also supposed to rename the plugins directory in the public_html directory also?