User-agent: * Allow: / # Prevent native HCL URLs from being crawled. Rule defined by Martin Dejnicki. Disallow: /shop/Product* # Prevent site search URLs from being crawled. Rule defined by Martin Dejnicki. Disallow: /shop/SearchDisplay*