CI: Remove cache where not needed

The specialized Docker images don't profit from using the portage tree
cache. Thus, this commit removes the usage of the cache for these jobs.

Signed-off-by: Adrian 'nex' Schollmeyer <adrian.schollmeyer@fem.tu-ilmenau.de>
3 jobs for gitlab-ci-images in 3 minutes and 40 seconds (queued for 8 seconds)
latest