Symfony impersonating user on different host

Question

I'm trying to impersonate a user from my application which is on a different host.

The admin user is on admin.example.local, and the user is on: example.example.local

Now I found some other questions which required to take some of the same actions but none which have been across hosts.

My security.yml

security:
    providers: 
        app_users: 
            entity: {class: MyApps\AppBundle\Entity\User, property: email}
        admin_users:
            entity: {class MyApps\AdminBundle\Entity\User, property: email}
    firewalls
        app:
            anonymous: ~
            host: ^(app)\.example\.(com|local)$
            guard:
                authenticators:
                    - app_users.security.login_form_authenticator
            switch_user:
                provider: app_users
                role: ROLE_ADMIN
            provider: app_users
            context: my_users
        admin:
            anonymous: ~
            provider: admin_users
            host: ^(admin)\.example\.(com|local)$
            guard:
                authenticators:
                    - admin_users.security.login_form_authenticator
            switch_user:
                provider: app_users
                role: ROLE_ADMIN
            context: my_users

The login is working fine on both hosts, i'm able to login without any problems. However, when I try to impersonate a user from the app firewall, I have will be redirected, I will be authenticated, but I am authenticated without impersonating the given user.

Am I missing anything that I need to impersonate cross hosts?

Already checked Simular question