WP Content Crawler – Get content from almost any site, automatically!
With the help of the WP material Crawler WordPress plugin, you’ll be able to automatically import material from virtually any website into your WordPress blog.
WP Content Crawler is an all-in-one content scraping and aggregation tool that is both powerful and versatile. It is designed to extract material from virtually any website, providing you with an automated method to source and curate content for your own website. This plugin, which boasts a long variety of useful features, can help you save time and resources while also guaranteeing that you will never be without new content.
FOR WHAT IT CAN BE USED
- To browse news, blogs, and other content from all of your favourite websites in one location, create a personal website.
- Utilize it with WooCommerce to gather items from online stores.
- To earn money, gather products from affiliate programmes.
- To construct a test environment for your plugin or theme, gather posts.
- Create a collection of plugins, themes, programmes, and photos by gathering them from other websites.
- Keep an eye on rivals
- You have unlimited imagination. Internet content is abundant.
Before you buy, make sure you do the following:
- Use the plugin in the demo after watching the quick start video. To understand how to utilise the plugin, you may also view the other video guides. Additionally, the plugin comes with a tonne of guides that explain how to use it for various tasks.
- Follow the directions in Can I receive material from X site? to make sure the plugin can retrieve the information from the website you wish to crawl. FAQ.
- Ask us in the comments if you’re still unsure about whether the plugin can retrieve content from a particular website.
- If you have any questions, you can look in the FAQs. You can always ask us a question in the comments section if the answer is not there.
Save every post detail
Everything from the title to the excerpt to the text to the tags to the categories to the slug to the date to the custom meta to the taxonomies to the featured picture to the post images to the status.
To find the CSS selector for an element, simply click on it. Additionally, you can obtain alternative CSS selectors that you might find useful. There is no longer a requirement to exit your admin panel.
Crawl (scrape, grab, save) posts
Following configuration of the options, the plugin locates post URLs and scans them automatically in the background.
Recrawl (update) posts
Automatically recrawl posts to keep them continually updated. You can define an update interval, restrict the number of times a post can be changed, and disregard older posts.
You wish to remove previously crawled posts. They may be automatically deleted by the plugin.
You can specify how frequently URL collecting and post crawling events for a site should occur. For instance, you can conduct URL collecting five times every two minutes or save three posts every minute.
The target category is absent from your website. No issue. The target categories can be made by the plugin. The CSS selectors that find category names should only be defined. Even subcategories can be made for them.
Save slugs (permalink)
The posts’ permalinks can be customised. Using short codes, you may acquire the permalink from the target site, add custom text, and even make templates for the slugs.
By obtaining them from the target site or manually entering them, save taxonomy values. It’s now simpler than ever to save custom post type details.
Save posts into custom categories
There are custom categories for a custom post type. No issue. When creating the categories for the post, you can define custom category taxonomies that are used by the custom post type and choose those categories. Additionally, the plugin enables you to design unique categories.
Custom post meta
Anything can be saved as custom post meta. You can type the value directly or use a CSS selector.
Utilizing short codes, create post content, title, excerpt, list item, and gallery item templates. Additionally, you can use the options box to build templates for the values of each CSS selector.
Even if the target site contains post pages that are uniquely built from one another, you can still construct alternate selectors to obtain the data.
Find and replace anything
To find and replace anything, you can use regular expressions or plain text. Even the HTML for the page can be changed. You can make your own HTML components and write selectors to use them. Even picture URLs can be changed. You are in charge.
Has the target post more than one page? Fear not. You may also store paginated posts.
List type posts
Some websites produce posts that contain a list. You can take the list out of the post, make a template that should be used for each item on the list, or even flip the list.
Type your posts. It might be any post type that is supported by your WordPress installation, including posts, pages, products, and more.
The post’s links can be deleted. The links disappear after checking the appropriate item. That simple.
To help you remember things about the website, you can make notes for yourself. anything—CSS selectors, a TODO list, etc.
Set CSS selectors for the category and post pages that shouldn’t have empty values. You can receive an email notification when one of those selectors discovers a value that is empty.
Advanced HTML manipulations
Replace elements in response HTML, discover and replace attributes in elements, swap out attributes for different ones, remove attributes, edit an element’s HTML, and remove HTML elements.
Recognize when an issue exists. The plugin will provide the error’s specifics for you to immediately correct.
You can add as many sites as you want to the plugin and activate them all.
Ready to translate
Using Poedit, you can translate the plugin into your native tongue.
Filters allow you to conditionally execute actions. If a certain word appears in one of a product’s attribute values, for instance, you can raise the price of that item. Action commands abound in filters. See the documentation for the commands.
Our vision and mission
We think everyone should have access to dependable, automatic, and powerful crawling capabilities. By making these skills available to everyone, not just developers, we hope to democratise this industry. With this in mind, we want to offer a plugin that you will adore and feel comfortable using. We made the plugin affordable and simple to use so that anyone can utilise it. We don’t use the features to boost sales alone. Our actions are directed toward the future. We always pay attention to your suggestions and make the necessary adjustments. We believe that enterprise-level caution should be used when developing WordPress plugins. To make sure the plugin is compatible with your server and WordPress environments and that you, our valued customers, receive the quality and dependability you deserve, we rigorously test it before each release. Currently, there are over 1700 automated end-to-end UI tests running across many different environments in the cloud.
How we solve these problems
Since we have been working on WP Content Crawler for about 4 years, we have encountered almost every scenario. We offer solid and dependable solutions to these issues by collaborating with our clients and paying attention to their demands. We think that instead of worrying about the intricate activities taking place behind the scenes, one could simply specify which site the information should be acquired from and what information should be retrieved from that website before beginning to crawl that site.
We offer comprehensive online documentation that explains how to utilise the settings as well as how to use them to achieve your goals in order to make it accessible to everyone. You might not always want to read the documentation. Additionally, we offer clickable interactive step-by-step instructions in the plugin. The interactive guides that walk you through specific tasks step-by-step can be opened at any time and from any stage you choose.
Testing practically any configuration is one of WP Content Crawler’s most remarkable capabilities. By doing this, when you enable automated crawling, there won’t be any unpleasant surprises. The mistakes associated with your settings are displayed during testing so you may make the necessary corrections before anything goes wrong.
Even we are unsure of the exact number of functions that WP Content Crawler contains. You can translate articles, spin posts, and even specify which fields need to be translated or spun if you do not want all of them changed. You can also automatically crawl, update, and delete the posts. Almost anything can be found and replaced. You can place information from the target post anywhere in the post by assigning it to a short code. Products from WooCommerce can be saved. For third-party plugins that we are not even aware of, you can save information. The plugin’s features are created so that when you use them, you feel in charge. To best suit your needs, we make them as versatile as we can. When creating new features, we constantly plan for the possibility that you’ll require a more sophisticated version of that feature. To constantly improve the plugin, we make sure that all of its features and source code are expandable and maintainable.