gitlab: ActiveRecord::ConnectionTimeoutError

Gitlab error on the webservice app, might be linked to the 500 we observe

I only saw this specific stack trace, this is very frequent when the GitLab is under load

It looks like rack_attack is trying to get the current rate limits settings and if they are not cached in memory ( https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/current_settings.rb#L46 ) it quickly exhaust the connection pool (might be a DoS vulnerability btw)

I couldn't figure out the value of IN_MEMORY_APPLICATION_SETTINGS and if we can enable it without breaking stuff

ActiveRecord::ConnectionTimeoutError (could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use):
  
app/models/concerns/cacheable_attributes.rb:19:in `current_without_cache'
app/models/concerns/cacheable_attributes.rb:63:in `rescue in current'
app/models/concerns/cacheable_attributes.rb:51:in `current'
lib/gitlab/current_settings.rb:60:in `uncached_application_settings'
lib/gitlab/current_settings.rb:42:in `ensure_application_settings!'
lib/gitlab/current_settings.rb:15:in `block in current_application_settings'
lib/gitlab/safe_request_store.rb:12:in `fetch'
lib/gitlab/current_settings.rb:15:in `current_application_settings'
lib/gitlab/throttle.rb:8:in `settings'
lib/gitlab/rack_attack/request.rb:71:in `throttle_authenticated_api?'
lib/gitlab/rack_attack.rb:92:in `block in configure_throttles'
lib/gitlab/middleware/read_only/controller.rb:50:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/same_site_cookies.rb:27:in `call'
lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:21:in `call'
config/initializers/fix_local_cache_middleware.rb:11:in `call'
lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:74:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'

Discussed here https://gitlab.com/gitlab-org/gitlab/-/issues/220879

The fix might not be as simple as to increase the pool size because this bug might be the consequence of an underlying problem in the ruby code

See https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/current_settings.rb#L60