How to Fetch All Users from AD LDAP in Laravel Without Timeout or Memory Issues?

I’m currently working on a Laravel 10 application that integrates with Active Directory (AD) using the AdLdap2 package and PHP’s LDAP functions (PHP 8). However, I’m facing challenges when trying to fetch all users (over 20,000) from the LDAP server.

Here are the two methods I’ve tried:
Method 1: Using AdLdap2 Package

private function handleAdLdapUsersEx($provider): void
{
    try {
        $pageSize = 200;

        if ($this->searchBase->filter) {
            $provider->where($this->searchBase->filter);
        }

        $pagedUsers = $provider->select('distinguishedname', 'givenname', 'name', 'objectguid', 'samaccountname', 'userprincipalname', 'mail')
            ->whereEnabled()
            ->limit($pageSize)
            ->paginate($pageSize, $this->currentPage);
        
        dump($pagedUsers->count());
        
        if ($pagedUsers->count() > 0) {
            collect($pagedUsers->getResults())->chunk(100)->each(function ($chunkedUsers) {
                $this->handleBulk($chunkedUsers);
                unset($chunkedUsers);
                gc_collect_cycles();
            });

            if ($pagedUsers->count() === $pageSize) {
                ImportUsersJob::dispatch($this->searchBase, $this->ldapConnector, $this->ldapConfig, $this->currentPage + 1);
            }
        }
    } catch (\Exception $e) {
        dd($e->getMessage());
    }
}

In this method, I set a limit for pagination, even with limit and pagination I am getting all the records in one page, but I’m still experiencing timeouts. Setting public $timeout = 600; works, but I’d like to avoid hardcoding a timeout.

Method 2: Using PHP LDAP Functions

private function handleAdLdapUsers($provider, $ldapConnector): void
{
    try {
        $usersFetched = 0;
        $lastUserEntry = null;

        $ldapConnection = ldap_connect($provider->getConnection()->getHost());
        ldap_set_option($ldapConnection, LDAP_OPT_PROTOCOL_VERSION, 3);
        ldap_bind($ldapConnection, $ldapConnector->getUsernames(), $ldapConnector->getPassword());

        $filter = !empty($this->searchBase->filter) ? $this->searchBase->filter : "(objectClass=*)";

        $result = @ldap_search($ldapConnection, $provider->getDn(), $filter, [], 0, 1);
        $firstEntry = ldap_first_entry($ldapConnection, $result);

        while ($firstEntry) {
            $attributes = ldap_get_attributes($ldapConnection, $firstEntry);

            $users = $provider->select('distinguishedname', 'givenname', 'name', 'objectguid', 'samaccountname', 'userprincipalname', 'mail', 'usncreated')
                ->whereEnabled()
                ->get($attributes);

            if ($users->count() > 0) {
                $this->handleBulk($users);
                $usersFetched = $users->count();
            } else {
                break;
            }

            $lastUserEntry = ldap_next_entry($ldapConnection, $firstEntry);
            $firstEntry = $lastUserEntry;
        }

        ldap_unbind($ldapConnection);
    } catch (\Exception $e) {
        dd($e->getMessage());
    }
}

This method returns a "Sizelimit exceeded" warning and only fetches 1000 records. I suppressed the warning with @ldap_search, but I still need a way to fetch all users (potentially over 50,000) without hitting size limits or running into memory issues.

How can I modify my queries or pagination to handle fetching all users efficiently?
Are there best practices for dealing with large datasets in Laravel when querying LDAP?
Is there a way to configure the LDAP server to allow fetching more than the default size limit?

Any help or guidance would be greatly appreciated!

You need to sign in to view this answers

Related Post