pbootcms屏蔽蜘蛛robots设置 每天蜘蛛抓几万 差点把服务器抓崩溃了
User-agent: *
Disallow: /ad*
User-agent: MJ12bot
Disallow: /
User-agent: YisouSpider
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-SI
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SemrushBot-CT
Disallow: /
User-agent: SemrushBot-BM
Disallow: /
User-agent: SemrushBot-SEOAB
Disallow: /
user-agent: AhrefsBot
Disallow: /
User-agent: DotBot
Disallow: /
User-agent: Uptimebot
Disallow: /
User-agent: MegaIndex.ru
Disallow: /
User-agent: ZoominfoBot
Disallow: /
User-agent: Mail.Ru
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: ExtLinksBot
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: Researchscan
Disallow: /
User-agent: DnyzBot
Disallow: /
User-agent: spbot
Disallow: /
User-agent: YandexBot
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: SemrushBot-SA
Disallow: /
User-agent: SemrushBot-BA
Disallow: /
User-agent: SemrushBot-SI
Disallow: /
User-agent: SemrushBot-SWA
Disallow: /
User-agent: SemrushBot-CT
Disallow: /
User-agent: SemrushBot-BM
Disallow: /
User-agent: SemrushBot-SEOAB
Disallow: /
如果您的问题还未解决可以联系站长付费协助。
有问题可以加入技术QQ群一起交流学习
本站vip会员 请加入无忧模板网 VIP群（50604020） PS:加入时备注用户名或昵称
普通注册会员或访客 请加入无忧模板网 技术交流群（50604130）
客服微信号：15898888535
声明：本站所有文章资源内容，如无特殊说明或标注，均为采集网络资源。如若内容侵犯了原著者的合法权益，可联系站长删除。
