Allow or Deny Search Engines by Using Robots.txt File

What is Robot.txt? 
Web site owners use the /robots.txt file to give instructions about a website to web robots, and this is called The Robots Exclusion Protocol. Website administrator is using robots.txt File to Allow or Deny Search Engines. If you have portions of a website that you do not wish for search indexes to see, you can protect them with a “robots.txt” file dictating which search engines are allowed or disallowed from seeing specific folders/files.

There are many options which you can specify in a robots.txt file to explicitly deny or allow specific search-bots to index certain folders or files. The simplest robots.txt file uses two rules:

1. User-agent: the robot the following rule applies to
2. Disallow: the URL you want to block
   
   These two lines are considered a single entry in the file. You can include as many entries as you want. You can include multiple Disallow lines and multiple user-agents in one entry. 

Two important considerations when using /robots.txt:

1. Robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
2. the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use. So don't try to use /robots.txt to hide information.

Can I block the Bad Robots?
In theory YES, in practice, NO. If the bad robot obeys /robots.txt, and you know the name it scans for in the User-Agent field. then you can create a section in your /robotst.txt to exclude it specifically. But almost all bad robots ignore /robots.txt, making that pointless.



DDoS Attack via Bad Robots
If the bad robot operates from a single IP address, you can block its access to your web server through server configuration or with a network firewall. In DDoS situation, the robot operate at lots of different IP addresses (hijacked PCs that are part of a large Botnet), to generate attack load to your server by just simply scanning your website. It will make the entire web server slow or stop with little of bandwidth involved. It's consider as level 7 of DDoS attack, the application attack.

The easiest solution is to use an advanced firewall to automatically block on all these IP addresses that make many connections. To learn more about our DDoS protection solution, please visit http://www.everworks.com/Services/DDoS

Please find below for some other articles that explains how a robots.txt file works, as well as how to and how to configure it for your website.

• http://www.robotstxt.org/
• http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=40360
• http://en.wikipedia.org/wiki/Robots_Exclusion_Standard

Read More

Business Email - Step 4: Email Options

We recently published our Business Email series to help you with your business email management strategy. As each post is posted up, we will dive deeper into each chapter to discover a better business management strategy.
Catch up on the previous chapters here: Step one, two, and three.

Knowing which email, hosting email, and requirements doesn't just stop there!

Find out what is the easiest solution available for you. Here is an easy breakdown of what you need to know most importantly.

EVERWORKS offers a simple all in one email solution that services almost every major need.

Here we have a breakdown of EVERWORKS's email offerings:

BasicWorks, MidWorks, and PremiumWorks suggests what the name actually means. Choose what is best for you! (Still not sure what to choose? Worry not, drop us an email and we will gladly help you out!)

This concludes our Business Email series! Hopefully this has given you plenty insight on how to manage and choose email. Stay tuned for more!

Tell us what you think:

1. What are your thoughts on the options given?
2. Between BasicWorks, MidWorks, and PremiumWorks which would you choose and for what purpose?

Follow us through our feed to follow our future updates through email and more.

Option photographed by Hope for gorilla from Flickr | Resourced from Rackspace

Read More

Business Email - Step 3: Email Requirements

Over the last few weeks we have covered in our Business Email Series: Hosting Email, and Which Email? This week we take a look at creating your email system requirement list. A breakdown of everything you may need now or in future. Breaking it down onto a list of requirements better prepares you for what are your immediate needs and gives you good control with anticipated growth requirements.

Begin identifying requirements by:

1. Auditing your system for current usage requirements
2. Polling current users for feedback on the current system
3. Review current email regulations in your country, or region to better protect your email rights

As you complete these steps, here are several tips that will help you along the way.

Talk to the right people: A person on the field, compared to an in-house employee might have a very different idea to what and how email should be efficient or useful. That is why it is important to gather information from a wide array of individuals with different backgrounds, experiences, and expertise to gather insight that would generally give everyone a good system to use.

(Having a tough time with internal surveys? Consider using Survey Monkeys to ease up the process of asking a large audience questions that you need answers to.)

Gather and analyze relevant data: Determining logs, failures, crash reports, and system outages to better improve your current systems, to meet the growing demands of your company will better improve your email response service team's ability to respond to an inquiry. Finding problems and solving it will prove to be vital minutes in revenue and productivity time earned.

Security: As covered before, advantages and disadvantages of free email vs hosted email, security is a major concern. Obvious security protocols are great, but when it comes to a greater need of security emails should be kept only for the eyes of the intended recipient. Ultimately it saves you greater hassle, and potentially large sums of money!

Tell us what you think:

1. Does breaking down what you need into a compiled requirements list help?
2. What other advice would you add to help other readers?

Follow us through our feed to follow our future updates through email and more.

Mission Impossible photographed by Alex Turton from Flickr | Resourced from Rackspace

Read More

Portal To The Future

A leap into the future, at least what seems to be the future. Just what is in store for all of us is mind boggling. Just imagine the possibilities that could readily shape our dreams, from potentially building a bridge into space, or delving deeper into the virtual realms created by numbers and powered by computers.

What is growing right now right at our doorstep? Organic human interaction has always been the goal, where technology would harmoniously function with human daily activities; where technology would no longer be considered a burden, but apart of everyday life, helping us to perform tasks to move the human race forward.

Shift in the way we interact with machines
Right now the known method of interaction is when we use an input device that the machine can recognize. At least that was the known method until the introduction of the iPhone in 2007. Where natural, organic human movements interacted with the machine. Now with the introduction of new technology where humans can interact with the virtual space with organic movements seem to be taking light.

Leap Motion looking to launch in July, and by placing a monitor in front of your eyes, Google Glass, the world is embracing technology and machines like it has never seen before. Social interaction is taking a twist back to reality, interacting with people and how they use it in everyday life.


Here comes exciting new tech.

Tell us what you think:

1. How will this adapt to the Malaysian atmosphere?
2. Where do you think technology is leading us to?

Follow us through our feed to keep up with us and posts similar to this in future.

Google Glass Image | Watch Image Pebble | Resourced from Smashing Magazine 

Read More