Adding item to listbox from another listbox taking way too long

I have two list boxes, one has a list of available users (about 1500 items) and the other is blank.
There is a button to add the seleted items from the first list box to the second one.
Both of these listboxes are within an update panel and for some reason it takes over 20 seconds for the items to show up in the second listbox.
Is this normal?  Is there an alternate, faster way of doing this? 
...