The 20% VAT is on the buyers premium, not the sale price, so if the boat sells at £10,000, there will be £2000 premium + £400 VAT. Price to pay £12,400.
I go to a lot of auctions and although 20% is a bit high, it's not way out. Given that they will also be charging a sellers premium, usually at the same rate, that £10,000 boat will make the auction house £4,800. It makes one wonder what they do to earn that money.