can Ostiary Thrull ability tap creature with protection from white?

since the ability of Ostiary Thrull requires white mana to cast, can it target pro white creatures?

Yes. The cost of the ability doesn't matter, what matters is the colour of this source - in this case, black. It can tap something with prowhite, but not problack.

but when you cast a spell doesnt the color of the spell is determined from colors of mana in cost? so when you cast an ability isnt it the same?

With abilities, you look at the color of the source.