Think about the links that you have built before. If all of them are coming from quality sites and the rankings of websites are not affected, then stop worrying and keep working in the same direction.