I discovered much faster way to speed up docker image creation.
Just use bundle package and all current gems will be copied into vendor/cache. When bundle install runs it discovers cached gems and runs much-much faster.