NSE proposal: New Rules


Version 1: http://seclists.org/nmap-dev/2010/q2/952
Version 2: http://seclists.org/nmap-dev/2010/q2/966


Table:
------
1) Introduction
2) More NSE Rules Background
3) Motivation
4) New NSE scripts details
5) New Rules
6) The Action function
7) Changes on Nmap/NSE code
8) Conclusion


1) Introduction:
----------------
In this proposal we will try to introduce new rules which can be used
by a new class of NSE scripts. 

The current NSE rules are:
o hostrule: specifies that the script should only run once against the
host IP.
o portrule: specifies that the script should only run against the ports
list of the current target.

The limitation of these rules is that the scripts will run only against
a single host IP (hostrule or portrule), and this host is provided by
Nmap which has already done a bunch of operations to get this information, 
so as an example, we can't run scripts against other targets which can be
networks or domain names that do not depend on Nmap previous operations.
* Network targets can be useful for host discovery and broadcast operations.
* Domain name targets can be useful to do DNS brute force operations.
Another limitation is that since NSE scripts depend on host and port
tables information they must run after Nmap scan phases.

A simple example:
* Script: dns-zone-transfer.nse
* Portrule: portrule = shortport.portnumber(53, 'tcp')
Modify the dns-zone-transfer.nse script and add another rule to let the
script run against the domain name to discover new targets. The current
script will only run when Nmap finds a DNS server, so with a new added
rule that script will run directly and does not depend on open ports and
can find new subdomain targets for Nmap, in other words specify a domain
name as a target and with the use of the results of this script, Nmap
will scan all the newly discovered subdomains and hostnames.


Please note that this is a first proposal about new rules. There will 
be other proposals to complete the New scripts class definition.


2) More NSE Rules Background:
-----------------------------
At present there are only two types of scripts, host and port classes
scripts. Each script has its own rule and action.

o The action function is the heart of an NSE script [1].
o The rule is the Lua function which will be evaluated by NSE to true or
  false.

The script's action will only be executed if its rule evalutes to true.

NSE Rules:
o Host rule:
"Specifies that the script should be run only once against a target
IP and only if the given conditions are met." [1]
The Host rule accepts a host table which contains the IP or Hostname of
the target as an argument.

o Port rule:
"Governs which ports of a target the scripts may run against" [1].
The Port rule accepts both host and port tables as arguments or any TCP
or UDP port in the open, open|filtered, or unfiltered port states [1].


3) Motivation:
--------------
1. Network operations:
We can need scripts that perform different network operations in order
to discover more hosts. In general these scripts will run once per scan
and can feed Nmap with new targets, but we can have other use cases.

 o Use broadcast and multicast protocols as suggested in this thread
   http://seclists.org/nmap-dev/2010/q1/883

 o We can also use AS entries to get IPs as suggested in this thread
   http://seclists.org/nmap-dev/2010/q2/101

 o Perform DNS discovery: brute force, zone transfert etc.

 o Perform Whois requests.

 o Any other operation which can take a network or a simple input that
 can be used to produce a list of Nmap targets.

 o Scripts that run after Nmap scans to collect/report results.

 o Scripts which want to use Nmap/NSE API.


4) New NSE scripts details:
-------------------------------
o The new scripts can run at different times:
  * Before any scanning: pre-scan scripts.
  * After Nmap scans: post-scan scripts.

Before/after NSE hostgroup scan can be emulated by the current NSE
dependency system.

o Pre-scan scripts can be used for host discovery like DNS scripts. We
can also have other cases of use like scripts which simply want to use
Nmap/NSE/Lua API to do cool tricks.
e.g: Ron's California Vanity License plate script :-)
http://www.skullsecurity.org/blog/?p=723

o Post-scan scripts: can be used to report data or to perform other
operations and take advantage of the Nmap/NSE/Lua API.
e.g: A script which collects all the whois hostrule script contact 
details results and report the unique ones.

o Since scripts can run at different times, then we need different rules.
 * prerule -- this rule will be used for pre-scan scripts.
 * postrule -- this rule will be used for post-scan scripts.

o The new scripts will be identified by the presence of these new rules
and by their categories.

o Scripts must use a new NSE API to push new targets.

o Hostrule and portrule scripts can have the ability to add new targets 
using the new NSE API, but perhaps we need a new Nmap option to allow 
this behaviour.

o Any target filtering will be done by Nmap. The new scripts must 
care only about their arguments.

o Each script can have its own arguments like any other script.
  e.g: script.arg1=x, script.arg2=x ...

o Scripts will use new API functions to access network interfaces 
like eth0, and mac address source etc.

A big point which will be discussed in the next proposal:
If NSE host discovery scripts will expand the Nmap target list or
narrow it down like the ping scan ?
Please keep in mind that this point will be discussed in a next
Nmap New Targets proposal.


5) New Rules:
-------------
Since we can have different scripts which can run at different times or
multiple times as stated in the [New NSE scripts details] section,
the imprortant question will be: When should the scripts run ?
For this we have proposed:

o Different rules funtions:
 * prerule -- this rule will be used for pre-scan scripts.
 * postrule -- this rule will be used for post-scan scripts.

o The new rules in most cases will evaluate to true.

o The new rules will be evaluated when the user specifies their
category with the --script option.

o The new rules will facilitate the addition of the new host discovery 
and scanning scripts.

e.g:
prerule = function() return true end
postrule = function() return true end


6) The Action function:
-----------------------
o Since new NSE scripts can run at different times, the action function
must know when it has been called, because some scripts can have
different behaviours especially when they have multiple rules.
e.g: A whois script which have a prerule and a postrule rules
will run at the beginning and sets a registry value to tell the whois
hostrule script to save the results onto the registry, after that the 
first whois script will run at the end (postrule) and collects/report 
the final results (e.g: unique contact details). Having two different
scripts, a pre-scan one and a post-scan script to do this seems to me  
rediculous.

o This feature of a script which can have multiple rules must not be
abused, if the action code differes a lot from when the script is in the
pre-scan mode and in the post-scan mode, then the best solution is to
have two different scripts each one with a rule.


Solutions to support scripts with both prerule and postrule:
o Add a SCRIPT_TYPE or RULE_TYPE field to the script thread's global table
that is the rule type of the running script: "portrule", "hostrule", 
"prerule", "postrule". The action function can then multiplex based on this
global field.
e.g:
  action = function()
    if SCRIPT_TYPE == "prerule" then
      ...
    elseif SCRIPT_TYPE == "postrule" then
      ...
    end
  end

o The rule function can return an optional second value which is the 
action function to run. If they only return a true boolean, then use the
global action function. This allows our current scripts to remain untouched
and more complex scripts can still use this new functionality.


7) Changes on Nmap/NSE code:
----------------------------

o Since there will be three different script scan phases then new scan types
must be defined. NSE needs also to know on which phase it is to activate the
correct rules.

NSE Scan Phases:
 * SCRIPT_PRE_SCAN: phase for prerule scripts, before any Nmap scan.
 * SCRIPT_SCAN: normal NSE scan for hostrule and portrule scripts.
 * SCRIPT_POST_SCAN: phase for postrule scripts, at the end of an Nmap scan.

Other changes:
o A new argument to the script_scan() to indicate the current script scan
phase, this argument will be assigned to the current_scantype variable.
e.g:
void script_scan (std::vector<Target *> &targets, int scantype)
{
  o.current_scantype = scantype;
...
}

script_scan(Targets, SCRIPT_PRE_SCAN); // new call of script_scan()


o A new string argument will be passed to the main NSE function, this string
is the result of the scantype2str(o.current_scantype) call.
The main NSE function can then setup prerule, hostrule, portrule and postrule
scripts according to this argument.

e.g (NSE main function):
local NSE_PRE_SCAN  = "NSE_PRE_SCAN";
local NSE_SCAN      = "NSE_SCAN";
local NSE_POST_SCAN = "NSE_POST_SCAN";
...
return function (hosts, scantype)

  -- activate prerule scripts
  if (scantype == NSE_PRE_SCAN) then
    ...

  -- activate hostrule, portule and versionrule scripts
  elseif (scantype == NSE_SCAN) then
    ...

  -- activate postrule scripts
  elseif (scantype == NSE_POST_SCAN) then
    ...
  end

  -- run scripts

end


8) Conclusion:
--------------
The addition of these new rules will let the scripts take different 
targets not only Nmap IP hosts or port lists, and will offer them
a way to run before or after Nmap scan phases.

This idea about a new class of scripts was first brought by Ron [2].

[1] http://nmap.org/book/nse-script-format.html
[2] http://seclists.org/nmap-dev/2010/q1/883