Java app increasing db connections every 10 mins by +8 connections

I have this situation … I have AWS and there I am running EKS (v1.31).. aside from EKS I have a standalone EC2 where I have postgres (PostgreSQL 14.13 (Ubuntu 14.13-0ubuntu0.22.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit) installed …

to this postgres I am connecting from my JAVA SpringBoot (Hikari) application .. the only connection config on the app side is

hikari:
      max-lifetime: 600000

The app is deployed via helm charts and ArgoCD to the EKS …

The helm chart values.yaml

    replicaCount: 1
    image:
      repository: *censored*
      pullPolicy: IfNotPresent
      tag: main-9610330a
    nameOverride: ""
    fullnameOverride: ""
    service:
      type: ClusterIP
      port: 8080
      healthCheckPort: 8080
    resources:
      limits:
        cpu: 500m
        memory: 768Mi
      requests:
        cpu: 200m
        memory: 512Mi
    podAnnotations: {}
    securityContext: {}
    podSecurityContext: {}
    nodeSelector: {}
    tolerations: []
    affinity: {}
    config:
      env:
    secrets:
      enabled: false
    ingress:
      enabled: false

values.dev.yaml

    ingress:
      enabled: true
      ingressClassName: nginx
      hostname: apigw-dev.example.com
      tls:
        - secretName: wildcard-cert 
          hosts:
            -  apigw-dev.example.com
      annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: 20m
        nginx.ingress.kubernetes.io/server-tokens: "false"         
        cert-manager.io/cluster-issuer: letsencrypt-cluster-issuer
        nginx.ingress.kubernetes.io/use-regex: "true"
        # enable CORS for subdomains
        nginx.ingress.kubernetes.io/enable-cors: "true"
        nginx.ingress.kubernetes.io/cors-allow-origin: "https://app-dev.example.com"
        nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, PUT, DELETE, OPTIONS"
        nginx.ingress.kubernetes.io/cors-allow-headers: "Origin, X-Requested-With, Content-Type, Accept, Authorization"
        nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
      hosts:
        - host: apigw-dev.example.com
          paths:
            - path: /business/[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}/logo
              pathType: ImplementationSpecific
              backend:
                service:
                  name: core.backend
                  port:
                    number: 8080
    config:
      env:
        .... some env variables are here .... 

The problem I am facing is, that in /var/log/postgresql/ files on the EC2 postgres server (ssh into the server) I can see plenty of these errors:

LOG: could not receive data from client: Connection reset by peer
FATAL: sorry, too many clients already

little bit of stats:

File                            Total Rows   Connection Reset Count  Too Many Clients Count
postgresql-14-main.log          5232         1589                    3064
postgresql-14-main.log.1        61           0                       0
postgresql-14-main.log.2        11496        3978                    6965
postgresql-14-main.log.3        19184        6293                    12566
postgresql-14-main.log.4        42705        2768                    0
postgresql-14-main.log.5        116          65                      0
postgresql-14-main.log.6        72           3                       0
postgresql-14-main.log.7        1430         1250                    0
postgresql-14-main.log.8        2881         2745                    0
postgresql-14-main.log.9        2883         2567                    0
postgresql-14-main.log.10       2783         2614                    0

and it seems that exactly every 10 mins new 8 connections are created … until it reaches the limit of maxConnections = 100 (set in pg config)

any reason what could this be?

You need to sign in to view this answers