DONE: o Augment the configure script to list unmet dependencies. Currently, configure works just fine without a C++ compiler installed, but make generates an error. The configure script should be able to detect this. Also, a list of features that are/are-not available would be nice at the end of the script, so folks can see that they've e.g. missed the OpenSSL dependency. o Add parallel IPv6 reverse DNS support (right now we use the system functions). o [Ncat] This may sound ridiculous, but I'm starting to think that Ncat should offer a very simple built-in http server (e.g. for simply sharing files, etc.) And maybe a simple client too. (Done via --lua-exec and the httpd.lua script shipped with Ncat) o INFRASTRUCTURE: Add IPv6 support to secwiki - We probably just have to designate a new IPv6 address for it and add it to Apache config. o [INFRASTRUCTURE] Improve our main web server http configuration to better handle high load situations and DoS attacks. As part of this, we may have to raise the max client limits. But then there is a risk of running out of RAM, which can be even worse. So we need to figure out a good balance. o Migrate web.insecure.org to a RHEL-6 derived distro (probably CENTOS 6, since Linode doesn't currently offer ScientificLinux images). o Actually, if we can wait until "second half of 2013", we might be able to jump straight to RHEL 7. And RHEL 5 support looks like it will go on for many more years for critical/security patches. o Maybe start with svn server, since we've had reports of our current one giving people unexpected password prompts. There is a thread about that at http://seclists.org/nmap-dev/2012/q2/17 o UPDATE on this - adding read-only rights (rather than no rights) to the root of the svn repo seems to have solved this problem. o Make Windows 8.1 VM with VS 2013 and do more testing of Nmap compilation/running o Make and test build on a newer OS X than 10.6 (10.10 was recently released) o Adopt an issue tracking system for Nmap and related tools. We should probably look at our needs and options and then decide on and either install it on our own infrastructure or use it hosted elsewhere. - David notes that Trac seems to work well for Tor -- see https://trac.torproject.org/projects/tor - One thing which can be nice is being able to interact with the system through email. Like for bugs people file on the Nmap package in Debian, I can just reply to the mail and it gets added in the tracker. - This is now live at http://issues.nmap.org/ o Update OpenSSL library to 1.0.1j o Our "make uninstall" should uninstall ndiff if it was installed too. We should probably do it in pretty much the same way we handle Zenmap (configure.ac, Makefile.in, and ndiff/setup.pl) o Web: We should probably distribute RapidSSL intermediate certificate on SecWiki so it is trusted even if browsers don't have that cert cached. Here's a page nothing the issue: https://www.ssllabs.com/ssltest/analyze.html?d=secwiki.org - We probably need to add an entry in apache conf after SSLCertificateFile which looks something like: SSLCertificateChainFile /etc/apache2/rapidssl.pem o The XML version of Nmap lists and describes the six port states recognized by Nmap near the top of the "Port Scanning Basics" section. That can be seen in the HTML rendering at https://nmap.org/book/man-port-scanning-basics.html. But in the man page (nroff) rendering, the list is missing and it just gives the title: "The six port states recognized by Nmap". UPDATE: Now the descriptions for each state appear in the man page, but the headings ("open", etc.) are missing. We should figure out why, and fix it. - The bug in the stylesheets means that (From Daniel): "if you have an element and it's followed by anything other than whitespace+CDATA (like " foo") then the remaining cdata or element until the next new element will be nroff-commented so this blah is ok, but this blah, is not ok because of the commaand this blah nmap -A is bad no matter how much whitespace intervenes" o Fix a segmentation fault in Ncat when scanned with the SSL NSE scripts. I was able to reproduce this on 2013-09-27 with latest SVN by running: Ncat: ncat -v -k --ssl -l localhost Nmap: ./nmap --script-trace --script '+ssl*' localhost -p 31337 This was initially reported by Timo Juhani Lindfors on the Debian bug tracker: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=724580 Henri notes: "I traced the latter back to openssl and opened a ticket there, which never got any reply... https://rt.openssl.org/Ticket/Display.html?id=2885&user=guest&pass=guest" o Investigate how we're ending up with OS fingerprints in nmap-os-db with attribute names like W0 and W8 when according to the docs they are only supposed to be W1 - W6 (and plain W). https://nmap.org/book/osdetect-methods.html#osdetect-w. See also http://seclists.org/nmap-dev/2013/q4/68. Need to determine how these are getting into the file (from Nmap itself or our integration/merge tools) and fix that then remove them from the file. o Integrate latest IPv4 OS detection submissions and corrections o We should improve the Windows build process for Ndiff, since it works differently now that it is modularized. To build the Nmap 6.45 release, we (as a temporary hack, not in SVN): - Added 'ndiff' to zenmap/setup.py 'packages' list in COMMON_SETUP_ARGS - Created a zenmap/ndiff subdir (empty) and copy ndiff/ndiff.py into zenmap/ before build. We should find a more elegant solution and check it into SVN. The fundamental issue is that the ndiff.exe we generate needs to be able to access the new ndiff.py module. Also, we need to make sure the -win32.zip Nmap distribution works properly. o [Zenmap] Combine parallel timed-out hops into one node in the topology view. http://seclists.org/nmap-dev/2012/q1/82 has a patch, however it doesn't handle the case of two or more consecutive timeouts. o If Nmap uses a "tcpwrapped" port to do fingerprinting on, OS detection might give false matches/results. Since it doesn't really matter which open port gets chosen, we should move onto another open port if we notice "tcpwrapped". o Implement an --exclude-ports option. See http://seclists.org/nmap-dev/2012/q1/275 o In an ideal world, Zenmap would not run out of memory and crash. And we already have an entry for improving Zenmap's memory consumption. But in the meantime, we should catch the error and present a more useful error message/explanation so the user understands the problem. This should reduce the number of out-of-memory "crash reports" we get too. See http://seclists.org/nmap-dev/2014/q2/298 o Provide an option to send a comment in scan packet data for target network. Examples: --data-string "Scan conducted by Marc Reis from SecOps, extension 2147" or --data-string "pH33r my l3eT s|<iLLz! I'll 0wN UR b0x!" o We should probably update our included libpcap. We currently include version 1.2.1 (we upgraded to that in April 2012) while the latest version on tcpdump.org is 1.5.3. We make minor changes to libpcap that we ship, and instructions for upgrading are in libpcap/NMAP_MODIFICATIONS. o Investigate report of Nmap ARP discovery using the wrong target MAC address field in ARP requests (it is correct in the ethernet frame itself). See this thread: http://seclists.org/nmap-dev/2011/q3/547 o Add randomizer to configure script so that a random ASCII art from docs/leet-nmap-ascii-art*.txt is printed. I think I'll start naming them leet-nmap-ascii-art-submittername.txt. o Add IPv6 subnet/pattern support like we offer for IPv4. o OK, we now have the subnet/pattern support, but not the two-stage model discussed below. So we added a separate task for that. o Obviously we can't go scanning a /48 in IPv6, but small subnets do make sense in some cases. For example, the VPS hosting company Linode assigns only one IPv6 address per user (unless they pay) and you can find many Linode machines by scanning certain /112's. And patterns might be useful because people assigned /64's might still put their machines at ::1, ::2, etc. o David says: "We need to design a new way to iterate over host specifications (i.e., different than nexthost). Because the new host discovery code is sometimes going to want whole netblocks and sometimes individual hosts. So I'm thinking of a two-stage model, where the iterator will received (parsed) specifications like AAAA::1/48, and then it can decide whether to further iterate that into individual addresses, or pass the block off to some specialized discovery routine." o Consider implementing RPC scan with ultra_scan or something else. Right now it is the only program using pos_scan. On the other hand, I'm not sure TCP RPC scanning is appropriate for ultra_scan. o When Ncat is compiled without OpenSSL, we should still accept the --ssl argument and just give an error message noting that SSL was not compiled in. This reduces confusion for users (e.g. http://seclists.org/nmap-dev/2013/q3/579) o We should update our OpenSSL Windows binaries from version 1.0.1c to something newer, like 1.01f o Web: figure out why autogeneration of nmap.org/nsedoc/ doesn't seem to be working. I think we had a cron job which was supposed to be doing it. - hb system was still running crontab files from old web vm in its rc.local. Fixed. o Add a W3C XML Schema Definition (XSD) for Nmap XML output. Keeping the DTD around is also helpful, but XSD is widely supported and could help improve support for Nmap XML in other tools. o We're going to discuss this on mailing list before deciding whether to 1) switch from DTD to XSD, 2) stick with just a DTD, or 3) try to support both. o Update copyright year to 2013 in the Nmap copyright header files o Update CHANGELOG for new release o New Nmap Release o Nping in ICMP mode (default) must not be checking the icmp IDs or returned packets or something, because if I have two separate 'nping scanme.nmap.org' running at the same time, each nping sees the replies from the other nping (as well as its own) and it screws up the timing stats too. o Process Nmap OS service detection submissions - New fingerprints + corrections - Last done November 2012: http://seclists.org/nmap-dev/2012/q4/222 o Process Nmap IPv6 OS detection submissions - New fingerprints + corrections o Process Nmap IPv4 OS detection submissions - New fingerprints + corrections - Last done in November 2012: http://seclists.org/nmap-dev/2012/q4/221 o Make Ncat reset the signal handler for SIGPIPE to SIG_DFL before execing a program with --exec and friends. A "broken pipe" error in a subprocess should kill the subprocess. Lack of default SIGPIPE handling is what prevents a trivial Lua chargen script--it loops forever after the socket disconnects because none of its writes fail. Cf. http://www.chiark.greenend.org.uk/ucgi/~cjwatson/blosxom/2009-07-02-python-sigpipe.html. o [Nping] In '-q' mode, Nping should keep the line giving the min/max/avg rtt times. That way people can avoid seeing each individual packet but still see the stats which are similar to what normal ping gives them. o [Nping] Remove the lines starting with 'Tx time' and 'Rx time' by default (and of course quieter modes), but leave them for cases at least one level of -v. o Nping/Nmap should probably show ICMP ping sequence values by default in packet trace mode. This would be nice for Nping since that is the default ping it sends and is the main way to distinguish the packets since the IPIDs are the same. o Complete migration away from Syn colocated machine - [Done - actually was already on web] Move submission CGIs to web - Make sure notification still works - [Done] Mailman - [Done] Install mailman software on web, including CGIs - Migrate mailing lists to web o Remove the -q/FAKE_ARGV stuff from Nmap, since I don't think people use that any more. o We should document Ron's sample script (https://nmap.org/svn/docs/sample-script.nse) in docs/scripting.xml so that new script writers know about it. - Decided to remove it instead. Justification: "It is a great idea, but nobody seems to use it (for example, there were no replies to usage inquiry here: http://seclists.org/nmap-dev/2012/q4/379). I think there are two main uses for this script, both of which are being served by other resources. 1) as a template for new scripts. Users instead seem to pick a script that is most similar to the one they want to write and start with that. 2) As a way to learn more about the format of an NSE script. Users instead seem to use our documentation (https://nmap.org/book/nse-script-format.html). So I'm deleting it for now. But if folks miss it, they're welcome and encouraged to say so on dev@nmap.org and we could consider putting it back and/or improving it" o Upgrade Mac Mini to Mac OS X 10.8 (Mountain Lion) and test building as well as testing usage of our normal builds (which we currently build on 10.6). o Make a branch from the 6.20BETA1 release (r30266) for new stable release, apply any important bugfix patches from the meantime and then release it after Thanksgiving as new Stable release. o [NSE] We may want to consider a better exception handling method -- one which doesn't require wrapping every I/O line in its own try function call. David says "Lua has an internal "exception handling" mechanism based on a function called pcall, which is implemented with setjmp/longjmp. You can wrap a function call in it and the function will return there whenever there's an unhandled error. Something based on that would be better [than the current system], I think." - This one is obsolete as the Lua 5.2 now lets you do a Lua yield across C function calls. o Add IPv6 support to Nping, including raw packet mode (hopefully sharing as much code with Nmap as possible, though Nping's packet code is a bit different), and also including echo mode server and client support. o Make sure we update everywhere relevant (e.g. refguide, etc.) to note the addition in Nmap of the Liblinear library for large linear classification (http://www.csie.ntu.edu.tw/~cjlin/liblinear/). It uses a three-clause BSD license: http://www.csie.ntu.edu.tw/~cjlin/liblinear/COPYRIGHT - David has added it to 3rd-party-licenses.txt - Fyodor moved it into the refguide o Consider including OpenSSL in our Nmap tarball - Need to check the size, etc. - OK, we're counting this as done because we took all the Win binaries out of the tarball and put them in an nmap-mswin32-aux svn directory which users check out to compile Nmap on Windows, and OpenSSL is included in this. o Update the Nmap CHANGELOG for latest improvements o Do an Nmap dev release. Last release was Nmap 6.01 June 22. o Update Nmap version number and auto-generated files for release. o Process latest Nmap OS submissions and corrections (IPv4 and IPv6). Last done (for IPv4 anyway) in February 2012. o Review and consider integrating Tomas Hozza's UNIX-domain socket support patch for nsock/ncat: http://seclists.org/nmap-dev/2012/q4/24. o Improve CPE coverage in OS detection DB from 84% to 90% (see CPE entry a ways down for more on this). o Process latest service detection submissions. They were last done in February 2012. o Integrate Henri's new kqueue/poll nsock-engines support. o If it is trivial to add, it would be nice if the "New VA Module Alert Service" also gave the Author field for NSE scripts so everyone knows which hero(es) wrote it. o Clean up the Nmap repo to remove some bloat we've allowed to creep in. Should do a more thorough search, but for now here are two obvious candidates: - Create publicly readable /nmap-mswin32-aux in svn - Files not needed for compiling Nmap itself (e.g. only needed for creating or including in Nmap packages), particularly including the vcredist files, should be moved to new /nmap-mswin32-aux - The /nmap-mswin32-aux files won't be included in Nmap tarballs either - Add the gtk, glib, etc. Windows dependencies to /nmap-mswin32-aux so users don't have to all install those in order to compile Zenmap and make Nmap packages. - move the nmap-private-dev/mswin32 stuff into /nmap-mswin32-aux - Update nmap-install.xml for new changes. Such as noting need to checkout this new directory for building packages, removing the need to install your own gtk, glib, etc. - [done] Remove the 5MB of XSL in nping/docs/xsl o Update our mswin32/OpenSSL to newest version (previous update was September 2010 to 1.0.0a). o Nmap should have a better way to handle XML script output. o done: https://nmap.org/book/nse-api.html#nse-structured-output o We currently just stick the current script output text into an XML tag. o Daniel Miller is working on an implementation: https://secwiki.org/w/Nmap/Structured_Script_Output o Update more web content in real time (or near real-time, or at least on an automated basis rather than requiring manual checkin and update). In particular: o NSEDoc generation o [done] SVN dir (https://nmap.org/svn/) should be removed and a redirect added to https svn server. o Maybe Nmap book building o Maybe the generated files in nmap.org/data/ o Update web.insecure.org so that rather than requiring us to build nsedoc on other machines, check it into svn, and then update svn on web, it is done by a script on web which could be run through cron (and potentially from a simple svn commit hook) to build them on the web server directly. - There are other similar things we might want to automate later, such as book rebuilding when the XML files are changed. o Investigate/fix potential routing-related issue. See emails from Djalal and others: http://seclists.org/nmap-dev/2012/q3/116, http://seclists.org/nmap-dev/2012/q3/4, http://seclists.org/nmap-dev/2012/q2/449 o Even without the --osscan-guess flag, Nmap should show the closest matches (if they pass our threshold) in the XML output. We omit them from the normal output in large part to encourage people to submit fingerprints, but that argument doesn't apply so well to XML output users. Normal output users who really want to see the Nmap guesses could still use --osscan-guess as before. o Change the interface of nmap.ip_send to take an explicit destination address. It currently extracts the destination from the packet buffer, which does not have enough information to reconstruct link-local addresses. See r26621 for a similar change that was made to Nmap internals. o [Zenmap] Install higher-resolution icons (at least 64x64 and maybe up to 512x512). Here is a screenshot of the current 48x48 icon on GNOME 3: http://seclists.org/nmap-dev/2012/q2/395. o Sean did Windows and Linux icons, and David did the Mac one. o [NPING] At least on my (Fyodor) system, I get errors like "READ-PCAP killed: Resource temporarily unavailable" with some commands. Example: # nping --tcp -p80 -c1 scanme.nmap.org Starting Nping 0.5.61TEST4 ( https://nmap.org/nping ) at 2012-02-16 17:52 PST SENT (0.3307s) TCP 192.168.0.5:42005 > 74.207.244.221:80 S ttl=64 id=23109 iplen=40 seq=1015357225 win=1480 RCVD (0.3524s) TCP 74.207.244.221:80 > 192.168.0.5:42005 SA ttl=51 id=0 iplen=44 seq=3197025741 win=14600 nping_event_handler(): READ-PCAP killed: Resource temporarily unavailable nping_event_handler(): TIMER killed: Resource temporarily unavailable [...] o [NPING] Nping should probably give you an error or warning when you do: "nping -p80 google.com" since it is ignoring the port specifier. The user probably wants to add --tcp. o Investigate why http pipelining so often doesn't work in NSE scripts, and often NSE ends up reverting to one request at a time. Scripts may not be using it correctly, and also we wish it were more transparent and there wasn't this big API divide between pipeline and non-pipeline. We just want it send requests as fast as it can, and get a callback when there's a response. Maybe the http library buffers them, or pipelines them, or blocks the http.get call until there's more room. It just seems to always degenerate to 1 request at a time. For example: sudo nmap --script=http-enum bamsoftware.com -p80 -d2 quickly (within a few seconds) gives: NSE: http-enum: Searching for entries under path '' (change with 'http-enum.basepath' argument) NSE: Total number of pipelined requests: 2081 NSE: Number of requests allowed by pipeline: 100 NSE: Received only 41 of 100 expected responses. Decreasing max pipelined requests to 41. NSE: Received only 1 of 41 expected responses. Decreasing max pipelined requests to 1. 100 may a wildly high number of requests to attempt to pipeline. And then something else probably goes wrong after it decides 41 is okay. - Related: Does caching work with pipeleined requests? We should make sure it does. [ OK, the main part of this todo item is done. Though there is a patch pending from Piotr which changes how pipelining works that is worth considering. We did fix the underlying pipelining bug, but (just as with most browsers), it isn't enabled by default. Also, it doesn't support caching. See http://seclists.org/nmap-dev/2012/q3/616. ] o Make Nmap from a clean start (e.g. after make clean or whatever, so it compiles everything) and research all the compile warnings to see which ones can be fixed/removed. Of course caution is needed to make sure we don't cause problems. For example, an unused variable on one platform might not be unused on another, so we can't just remove it. May have to surround it by ifdefs though. o Solve "spurious closed port detection" issue discovered by David: http://seclists.org/nmap-dev/2012/q1/62 . So we need to figure out what is going on here and then how to fix it. Note that this doesn't seem to happen when you do ICMP host discovery first (-PE), so it probably relates to the ACK packet that Nmap sends to port 80 on the target by default. o Add real headers for more protocol types in -6 -sO scan. Dario Ciccarone provided some packet captures for 0x00: hop-by-hop 0x2b: routing 0x2c: fragment 0x3c: destination (http://seclists.org/nmap-dev/2011/q2/1003). We also have examples of crafting some of these in FPEngine.cc. [Sean and David] o Investigate increasing FD_SETSIZE on Windows to allow us to multiplex more sockets. See Henri's email: http://seclists.org/nmap-dev/2012/q1/267 [James Rogers did some investigative work on this in July 2012, but we weren't able to find a great solution. Maybe we should investigate this more in the future, and also investigate other Windows socket APIs such as completion ports. ] o Finish sv-tidy - a program to canonicalize and tidy nmap-service-probes. o Check for the same reference (like $1) being used in unrelated fields (where related fields are the pairs (p, cpe:), (v, cpe:), (i, cpe:), (o, cpe:)). For example if we have v/$1/ h/$1/ it is a bug. o Check a list of common product names that should only appear in p//, not in i//. We still have entries that are like this: p/Foobar 2000 ADSL router/ i/micro_httpd web server/ that should rather be written this way: p/micro_httpd/ i/Foobar 2000 ADSL router/ o [Done] Check for e.g. i/French/ without :fr in cpe:/a, and vice versa. [Sean and David?] o Remove Nmap's --log-errors feature and make its behavior the default. A few notes: - Nmap should just ignore --log-errors if it sees it - Remember to remove it from the documentation o We should probably sort script output (for port output and host output) by script name or something so that it comes in a deterministic order. If the same three scripts produce output in two different scans, they should be listed in the same order. Right now the order can vary, at least for host output. [Sean] o Add a function such as --disable-arp-ping which prevents hosts from being automatically detected as 'up' just because they responded to ARP. Instead, Nmap will actually send the requested host discovery probes (ICMP ping packets, SYN packets, etc.) and only mark the host as up if it responds on an IP level. This is how machines are already treated if they're not on the local network (e.g. if ARP discovery is unavailable). This technique is a bit slower and more likely to miss hosts (e.g. if they're heavily firewalled) than ARP discovery, but the option is needed to handle local networks which use proxy ARP, which would otherwise cause all IPs to appear to be up. o We should add fields to the service submitter [James is working on this] (http://insecure.org/cgi-bin/submit.cgi?new-service) for the application name and version. o We also need to ensure all fields of /cgi-bin/submit.cgi have proper escapting to prevent possible reflected XSS attacks reported by Maxim Rupp (@mmrupp). The risk is low, if any, since we don't give authentication cookies for bad guys to steal, but is still better to properly escape. o If we get a chance, would be interesting to run our XSS-testing NSE scripts against this and see if they locate the problems. o Also, need to change the font family in there from "Lucida Grand" to "Lucida Grande"? Just a typo. And fix "WIkipedai". We should just spell-check all the output o Make Nmap 6.01 release containing (among possibly other little fixes) - Python upgrade - [done] Zenmap 10.7 hang fix (done in trunk) - [done] Zenmap crash when filtering hosts (done in trunk) - [done] get_srcaddr fix (done in trunk) o Upgrade Python on build machines to try and resolve Python 2.7 security warning (it doesn't affect us, but can worry users). See this thread: http://seclists.org/nmap-dev/2012/q2/621 o Fix get_srcaddr error happening on Windows XP o [Web] Add a page with the Nmap related videos we do have already - We have a page on Secwiki now: https://secwiki.org/w/Nmap/Presentations o Zenmap hang on OS X 10.7 o For many years, the Nmap man page and online documentation has had an "Inappropriate Usage" section which notes that "Nmap should never be installed with special privileges (e.g. suid root) for security reasons". And of course Nmap's official installer would never install Nmap that way. While one would thinks that would be enough, we might want to go even further and have Nmap detect when it is run suid and print a security warning. o Prepare release notes, web page, etc. o Do private beta release o Make the release o In Nmap XML output, osclass (OS Classification) tags should be children of osmatch (the human readable OS name line) rather than having Nmap deduplicate all the osclasses and put them in as siblings. But this change might break some systems which utilize Nmap XML output, so, along with this change, we need to introduce an option such as --deprecated-osclass-xml to return the old behavior. That option only needs to be documented in the CHANGELOG entry referring to this change, and it should note that we're likely to remove this option in a year or two. o Right now, when an IPv4 or IPv6 address seems bogus (such as 1.2.3 or 2001::0 in IPv4 mode), we give a fatal error and abort the scan. But since that might just be one bad target in a long list of hosts to be scanned, it is probably better to just print a warning and continue. Some sort of warning or host element should be included in the XML to explain what happened too. This should also happen if we're unable to resolve a DNS name. o In sv-tidy, check that used references start at 1 and are contiguous. If $1 and $3 are used but not $2, it's probably a bug. Maybe you can even find out how many there should be by inspecting the regular expression. o Raw scans from Mac OS X seems not to retrieve the MAC address or do ARP ping, except when scanning the router on an interface. For example, scanning 192.168.0.1-5 sends ARP pings to 192.168.0.1, but the normal four-probe combination to the other addresses. The "MAC address:" line appears in the output for .1 but not for the others. o To avoid Nmap memory usage bloat, find a way for NSE scripts to store information about a host which expires after Nmap is done scanning that host (e.g. when the hostgroup containing that host is finished). Right now scripts store such information in the registry and it persists forever. For example, a web spidering script/library could store information about the web structure and even page contents so that other scripts can use that information without spidering the target again, but ensuring that the memory will be freed after the hostgroup finishes so there is room to store the web information for the next group of systems. One idea would be to make a host.registry member which contains a registry specific to a specific target. Scripts could store temporary information there, but still use the global registry for information which must persist (e.g. to be used by postrules, etc.) o Add CPE support to IPv6 OS detection o Use BPF libpcap logic on Solaris 11, otherwise packet capture doesn't work at all. http://seclists.org/nmap-dev/2012/q1/613 o [NSE] host.os should not just be a list of strings which can contain human-readible strings and/or CPE info. It should probably be list of host.os tables which can contain: host.os[].name <-- human readible name host.os[].class[].vendor host.os[].class[].osfamily host.os[].class[].osgen host.os[].class[].devicetype host.os[].class[].cpe[] <-- array of cpe:/ strings So host.os[1].class[1].cpe[1] is the first CPE entry for the first classification of the first OS match for the target system. The host.os entry docs/scripting.xml would have to be updated too. o We should probably go through the nmap-os-db (and IPv6 version) entries and, where the fingerprint line specifies a service pack number (or even two of them), ensure that we have sp-qualified CPE entries like "cpe:/o:microsoft:windows_xp::sp2". Right now we sometimes include the qualification, and sometimes not. o This is best done with cpeify-os.py, if possible. o Zenmap no longer ads the installed module directory to its module search path because some distributors first install in a world writeable directory (like /tmp) and then put those files into their packages which they distribute to users. But this change can lead to Zenmap not working for users who install in nonsystem areas like their home directory (e.g. --prefix /home/fyodor) unless they have their PYTHONPATH set to find them. We should implement a solution, such as making sure Zenmap catches the missing modules error and suggest that the user set their PYTHONPATH or something. o Scans from Mac OS X tend to use raw IP packets rather than ethernet frames even on the local network because Dnet does not seem to be retrieving the routing table properly -- so the LAN doesn't even show up in --iflist. Patrik can reproduce this on all 3 of his MACs (OS X versions 10.7.3). Comparing the code in DNet route-bsd.c to Apple's own routing table code discovered by Patrik suggests that the Dnet code may be incorrect. o ssl-google-cert-catalog should not require that the user specify ssl-cert in order to run. Instead, they should probably both call a library which obtains the certificate (and caches it so that it doesn't happen twice if both scripts are run). In general, we want to avoid having any scripts tell the user "this script only works if you specify this other script too". If we really find we need that functionality, we should add a "strong dependencies" feature so that scripts can tell Nmap what other scripts they require. [Patrik did this by adding an ssl cert library] o Our targets-ipv6-multicast-slaac.nse should probably send the router advertisements with low priority to reduce the chances of any negative impacts on clients, if we're not doing that already. See http://lists.si6networks.com/pipermail/ipv6hackers/2012-March/000503.html. - Actually, I think we already do this. Marking as done. o Deal with the issue of timeouts happening too soon due to global congestion control in some cases. For example, if Nmap sends host discovery probes to two hosts, and one comes back extremely quickly, it can cause the global congestion control to use a very low timeout and cause the 2nd host (which doesn't have any host-based congestion control values yet) to timeout arguably too quickly. We should look at potential algorithm changes to improve this. David: I think I was wrong about the cause of this. Even when replies come back very quickly, the timeout is by default limited to 100000 microseconds, much higher than the straightforward calculation would give. What I think is really happening is that select is not working reliably on this platform (Solaris 10 x86). In the loop in read_arp_reply_pcap, pcap_select returns 1, then a pcap_next is done. Then pcap_select returns 0, but if I insert another pcap_next after that, the pcap_next finds another packet without blocking (the first time, anyway; after that it blocks). o Create CHANGELOG o Make stable release candidate branch o Make at least one more test release from the candidate branch o Write and send GSoC 2011 results email o Document the nsearg format changes made by Paulino (how you can preface an argument with a script to make it more specific, or make it general to apply to multiple scripts) o Rough drafts: o nmap-exp/calderon/refguide.xml o nmap-exp/calderon/scripting.xml o Relates to: o We should probably modify stdnse.get_script_args so that it first checks [scriptname].[argname] and then (if that fails) looks for [argname] by itself. This way people who are only running one script or who want to use the same value for multiple scripts that take the same argument can just give [argname]. But those who want an argument to only apply to a specific script can give [scriptname].[argname]. o Make the nmap.header.tmpl wording a little more generic so it more clearly applies to Ncat, Zenmap, Nping, etc. Then use templatereplace.pl to apply those changes to the code. [Fyodor] o Change Nmap copyright dates (in the file headers, etc.) from 2011 to 2012. o Get RPM staticly linking to libsvn (rather than dynamic linking) so that it isn't a requirement for installing the RPM. - We decided to just make nmap-update its own separate RPM so that it can dynamically link to libsvn without forcing that dependency on the whole nmap RPM package. - since the libsvn-devel package apparently only installs dynamic libs, we'll probably have to install it ourselves on the CentOS build machines. o Fix "BOGUS! Can't parse supposed IP packet" in packet trace of IPv6 packets. o Integrate latest IPv6 OS detection fingerprint submissions - In addition to the submission CGI submissions, some were emailed to Fyodor and David on Oct 21 o Integrate new service fingerprint submissions (we have more than 2,531 submissions in two files since 11/30/10) o Integrate new OS detection submissions (1,893 since 6/22/11) o Add options in configure script for users to specify where to find subversion lib/include dirs (like we do with our other library dependencies). See this mail: http://seclists.org/nmap-dev/2012/q1/37 -- David added --with-apr and --with-subversion o We need to fix the svn server so that Nmap committers can make branches from /nmap to /nmap-exp. We may need to add some sort of OPTIONS permission to the root directory or something, because they're getting errors like: $ svn cp https://svn.nmap.org/nmap https://svn.nmap.org/nmap-exp/branchname svn: Server sent unexpected return value (403 Forbidden) in response to OPTIONS request for 'https://svn.nmap.org' - Patrick also reported some other funny business related to svn mv'ing directories in email to Fyodor and David. o Give CPE visibility to NSE. - done by Henri o Document the new IPv6 OS detection novelty system in os-detection.xml o Do more thinking/researching/investigating the way our machine learning IPv6 OS detection system decides whether a match is perfect and/or how close the match is. Maybe our current system works well enough, we'll need to watch how it performs as we increase the DB size and collect/integrate more signatures. The goal is to: o Producing fewer way-off matches since it would have a way (like our current system) to decide how close the match really is o Doing a better job about printing fingerprints for matches with aren't close enough o Improve the "run Zenmap as root" menu item to work on distributions without su-to-root. We might even want to improve Zenmap so that it itself does not have to run as root, and just executes Nmap that way. Rather than not showing Zenmap as root on the Menu of non-working systems, it might be better to have it but let it give an error message (and then, perhaps, run as nonroot) so that users of those distributions are more likely to contribute a fix. We also might want to look at how the distributions themselves package Zenmap. o Consider changing Nsock so that it is able to take advantage of more modern interfaces to dealing with large sockets, rather than just select. Perhaps we should look at poll(), Windows completion ports, and some of the advanced Linux APIs. Select() limits us to descriptors no higher than FD_SETSIZE, and it may not performa all that well. We should do some benchmarking and decide on the interface to use for each platform. May want to take a look at libevent (http://www.monkey.org/~provos/libevent/) for inspiration. The libevent home page has some interesting benchmark graphs too. [Josh implemented poll as a SoC student, but it had problems with Nsock's architecture. O(1) lookups were becoming O(n) because of the nature of the data structures. It was slower in his benchmarks. Nsock would have change from a model of "loop over the event list, and check to see if the fd for each event is set," to one of "loop over the fd list, and see if there is a corresponding event for each. It is the "see if the fd is set" operation that's O(1) with select (it's FD_ISSET) and O(n) with poll (it's a traversal of a linked list).] o Henri added nsock-engines o Consider an update feed system for Nmap which let's people obtain the latest Nmap data files, such as NSE scripts/libs, nmap-os-db, nmap-service-probes, etc. o Note that some scripts require updated compiled libraries. We will need some sort of compatability system. o One approach is "svn up". Note that Metasploit uses that approach even for Windows by shipping .svn directories and an svn executable with the Windows installer. In taht case we might need to have a separate branch for each release that gets updated version/OS databases and scripts. o Another approach is a special feed system as is used by Nessus and OpenVAS. OpenVAS uses a script wrapper around rsync, or an HTTP download if that fails. o Colin's analysis of different methods: http://seclists.org/nmap-dev/2011/q2/821 o [NSE] Consider using .idl files rather than manually coding all the MSRPC stuff. The current idea, if we do this, is to have an application in nmap-private-dev which converts .idl files to LUA code for nmap/nselib. Consider adapting the pidl utility from Samba. o Drazen did some work on this during SoC. https://svn.nmap.org/nmap-exp/drazen/nmap-msrpc could get someone started. o We moved this out of the active section of the TODO because, while it is still a good idea and we'd welcome the change if someone wants to take it on, it isn't something that we are likely to make progress on unless someone steps forward. o Implement a solution for people who want NIST CPE OS detection results (we'll save version detection for a 2nd phase). Notes: David report on CPE for OS Detection: http://seclists.org/nmap-dev/2010/q3/278 David report on CPE for version detection: http://seclists.org/nmap-dev/2010/q3/303 Nessus has described their integration of CPE: http://blog.tenablesecurity.com/2010/05/common-platform-enumeration-cpe-with-nessus.html. Older messages about it: http://seclists.org/nmap-dev/2008/q4/627 http://seclists.org/nmap-dev/2010/q2/788 o [NSE] HTTP spidering library/script o We should probably modify stdnse.get_script_args so that it first checks [scriptname].[argname] and then (if that fails) looks for [argname] by itself. This way people who are only running one script or who want to use the same value for multiple scripts that take the same argument can just give [argname]. But those who want an argument to only apply to a specific script can give [scriptname].[argname]. o The code is in place now, we just need to document the feature. o Script review o Martin Swende patch to force script run http://seclists.org/nmap-dev/2010/q4/567 o applied o irc-info patch. http://seclists.org/nmap-dev/2011/q2/289. o applied o http-slowloris. http://seclists.org/nmap-dev/2011/q1/916. o Had some issues--never got to a state ready for integration o http-phpself-xss - Would need to be rewritten to use newer spider.lua. Added an item to incoming section of Nmap Script Ideas secwiki page. o Make new SecTools.Org site with the 2010 survey results. o Collect many more IPv6 OS detection training samples from users - Can start with nmap-dev, but will probably have to do an Nmap release too. o Integrate more NSE scripts, I think our review queue is getting pretty long. o Decide what to do with Henri's nsock-engines branch (/nmap-exp/henri/nsock-engines). o finish making nmap-update part of the nmap windows compile-time infrastructure o See if we can build just one project within a solution, rather than having special "with nmap-update" configuration. o Add homedir support to Nmap for the updater o Fix expiration date parsing on Nmap Windows for the updater o Updater: Make a missing nmap-update.conf nonfatal (perhaps doesn't even need to mention it). o Updater: Clean up the output messages (e.g. only print what user needs to see unless debugging is specified) o [Nping] The --safe-payloads option should be default (though we should keep it for backward compatability). We could then introduce --include-payloads for cases where they are desired. o A program to canonicalize and tidy nmap-service-probes. o Order of fields: m p v i d o h cpe:/a cpe:/h cpe:/o. o Check for duplicate templates (except cpe:). o Check for unknown templates. o Canonicalize delimiters (use // first, otherwise try in order | % = @ #). o Retain line breaks and comments. o Document IPv6 OS detection at https://nmap.org/book/osdetect.html o Script review: - New scripts from Paulino: http-wordpress-brute and http-joomla-brute, http-majordomo2-dir-traversal.nse, http-trace, http-waf-detect - http-methods patch. http://seclists.org/nmap-dev/2011/q1/936. - quake3-info. http://seclists.org/nmap-dev/2011/q2/172. - smb-os-discovery additional information. http://seclists.org/nmap-dev/2011/q2/276. - Outlook web address. http://seclists.org/nmap-dev/2011/q2/296. [probably not going to merge to Nmap trunk at this point, though it is good that the script is available for d/l for those who need it. ] o Fix reported (by many people) crash when trying to launch Zenmap on Mac OS X 10.7 (Lion). o Unless we get good arguments for keeping it, we should remove Mac OS X PowerPC support from our binaries. Apple stopped selling PowerPC machines in 2006 and they stopped making new OS releases available for PowerPC as of Snow Leopard (10.6) in August 2009. See this thread: http://seclists.org/nmap-dev/2011/q3/430 o Improvements to the Nmap multicast IPv6 host discovery scripts - Note that we hope to move them into core Nmap at some point, but would be good to improve them for now. - They should probably print the discovered IPv6 addresses, otherwise they don't actually give the user any information (despite doing their work) unless you give the newtargets script arg. This would be similar to the current behavior of broadcast-ping. - It might be nice if they gave the target MAC address and vendor when printing the discovered IPv6 information too. Daniel Miller wrote an initial patch for this (though we need to make sure it can handle (e.g. doesn't crash for) non-ethernet devices:http://seclists.org/nmap-dev/2011/q3/862. Our broadcast-ping script currently prints MAC addresses. - It is great that the scripts properly use a specific device when given the Nmap -e option, but they shouldn't require this. They should do something smart if no specific device name is given. Examples include performing on all compatable devices or trying to pick the best device. The all-devices appraoch may be the best, IMHO. That is how our broadcast-ping script works now. o Add anti-spam defenses to secwiki.com to stop the current onslaught of spam. An extention like ConfirmEdit (http://www.mediawiki.org/wiki/Extension:ConfirmEdit) may be a good choice. o Collect a bunch of IPv6 OS detection signatures from users, integrate them, and then when we have enough, re-enable OS detection results. o IPv6 OS detection working (when run on) Solaris and AIX - AIX 6.1 - iSeries / System p - AIX 7.1 - iSeries / System p - Solaris 10 - SPARC o We should consider splitting a 'brute' category out of the 'auth' category now that we have so many brute force scripts. I suppose users can already do "--script *-brute", but having its own category might still be nice. o IPv6 OS detection merge o [DONE] Initial branch working (nmap-exp/luis/nmap-os6) o [DONE] Implement the 2 remaining probes o [DONE] Disable the printing of matches (except maybe with debug on). We want more training examples first so that results are better. o [DONE] Merge to /nmap o Document Nmap CPE support in appropriate places (candidates: refguide, os detection book chapter, version detection book chapter, output book chapter). o Finish CPE support code - Escape certain values that can be inserted into cpe string through substitution, like cpe:/a:apache:httpd:$1 where $1 contains a colon. o Add advanced IPv6 host discovery features o Initially done using NSE by adding these scripts: targets-ipv6-multicast-slaac, targets-ipv6-multicast-invalid-dst, and targets-ipv6-multicast-echo o Initial IPv6 OS detection system (may not make it into stable though, but we want to at least have it working in a branch first.) - OK, it is working in nmap-exp/luis/nmap-os6 o Investigate a probe/response matching problem reported by QA Cafe Matthew Stickney and Joe McEachern of QA Cafe. See this thread: http://seclists.org/nmap-dev/2011/q3/227 o When our winpcap installer is run in silent mode (e.g. "winpcap-nmap-4.12.exe /S"), it seems to execute nmap.exe if that binary exists in the same directory. This leads to a cmd.exe window briefly poping up as Nmap displays its console help output. Moving the Winpcap installer into its own subdir and running it from there seems to fix this (because it then can't find nmap.exe to run), but it would be better to determine why this is happening in the first place and fix it. o Obtain Nmap data directory information from nmaprc at runtime rather than compiled in -- among other advantages this is needed to make relocateable rpm. [actually we ended up doing this without needing nmaprc for now] o Summer of Code feature creeper: o Ncat should probably have an --append-output option like Nmap does so that we can use -o without clobbering existing file. This would at least be useful for chat.nmap.org. o Change Zenmap bug reporter so that instead of an automatic submission system, we print a stack trace and request that the user send a bug report to nmap-dev. o [Ncat] Solve a crash that only happens on Windows when connecting with --ssl-verify and -vvv, for example ncat --ssl-verify -vvv www.amazon.com 443 The crash happens in the function verify_callback, when the function X509_NAME_print_ex_fp is called. Just commenting those two calls avoids the problem. By trying different combinations of debug print statements, I once got the message OPENSSL_Uplink(10109000,08): no OPENSSL_Applink This refers to a Windows dynamic linking issue: http://www.openssl.org/support/faq.html#PROG2 However I tried both including and changing the linker mode to /MD, and neither changed the behavior. Changing the flags from XN_FLAG_ONELINE to 0 seems to make the problem go away. o Integrate new OS detection submissions (We have about 1,700 submissions since 11/30/10) o Nmap should defer address parsing in arguments until it has read through all the args. Otherwise you get an error if you use like -S with an IPv6 address before you put -6 in the command line. You get a similar problem if you do "-A -6" (but "-6 -A works properly). This is a possible feature creeper task. o Ncat chat (at least in ssl mode) no longer gives the banner greeting when I connect. This worked in r23918, but not in r24185, which is the one running on chat.nmap.org as of 6/20/11. Verify by running "ncat --ssl -v chat.nmap.org" o IPv6 Neighbor Discovery-based host discovery (analog to ARP scan). o Investigate and document how easy it is to drop Ncat.exe by itself on other systems and have it work. We should also look into the dependencies of Nmap and Zenmap. It may be instructive to look at "Portable Firefox" (http://portableapps.com/apps/internet/firefox_portable) which is built using open source technology from portableapps.com, or look at "The Network Toolkit" by Cace (http://www.cacetech.com/products/network_toolkit.html). For Nmap and Nping, we may want to improve our Winpcap to load as a DLL without requiring installation. There is a separate TODO item for that. o The SCRIPT_NAME variable should not include the ".nse" in script names. Currently, it omits that for scripts in the DB, but includes it for scripts you specify based on their filename. See: http://seclists.org/nmap-dev/2011/q2/481 o If possible, Ncat, in listen mode, should probably listen on the system's IPv6 interfaces as well as IPv4. This is what servers like apache and ssh do by default. It might now be possible to listen on IPv6 by running a second ncat with -6, but that doesn't really work for broker and chat modes because you want the IPv6 users to be able to talk to IPv4 and vice versa. - This was partially implemented, but still doesn't seem to work in --chat mode. Can test against chat.nmap.org - Done. Tested on scanme with David & Fyodor on 7/18/11. o Right before the release, we could build Ncat portable and post it on https://nmap.org/ncat/. - Actually we did that for 5.59BETA1, which is good enough for now. o CHANGELOG updates [Fyodor] o [Ncat] Add new certificate bundle (ca-bundle.crt) since the current one is out of date. See http://seclists.org/nmap-dev/2011/q2/641. o Move these prerule/postrule script ideas to secwiki script idea page if appropriate (with a bit more details): o AS Number to IP ranges: http://seclists.org/nmap-dev/2010/q2/101 In progress. o DNS service discovery (Bonjour): http://en.wikipedia.org/wiki/Bonjour_%28software%29 Present as dns-service-discovery.nse. o Netbios Name Service Already present as broadcast-netbios-master-browser.nse? o DHCP broadcast requests Present as dhcp-discover.nse. o Postrules could be created which give final reports/statistics or other useful output. Like a reverse-index, which shows all the open port numbers individually and the hosts which had that port open (e.g. so you can see all the ssh servers at once, etc.) Admittedly you can do that pretty easy with Zenmap instead. Have a few of these: ssh-hostkey and upcoming creds-summary. o We could have a prerule sniffer script which uses pcap to sniff traffic for some short configurable amount of time and then adds the discovered hosts to the target list. Already present as targets-sniffer.nse. o We could have a script which takes traceroute results and adds them to the target list. Already present as targets-traceroute.nse. o [NSE] Add these ideas to secwiki script ideas page if appropriate (with a bit more details): o Windows system logs (like sysinternals' psloglist) o Services (like sysinternals' psservice) o A script (or modification to smb-check-vulns) to detect this MSRPC vulnerability: http://seclists.org/fulldisclosure/2010/Aug/122 o BasicHTML/XML parser library? For example, Sven Klemm wrote a script which uses libxml2: http://seclists.org/nmap-dev/2008/q3/0462.html. And here is one by Duart Silva using Expat: http://seclists.org/nmap-dev/2009/q3/1093. o Add detection of duplicate machines via IP.ID technique. Maybe I should use uptime timestamps too. Oh, and MAC addresses too. Our SSH host key script is useful for this as well. o Look into iplog ( http://ojnk.sourceforge.net/ ) -z option which is supposed to fool OS detection. o The software is no longer maintained, so we're not going to worry about it. The page says: "I am through working on this project. I will not be making any updates, and I will ignore just about all email about it. If anybody wants to take it over (for whatever reason), let me know" o [NSE] Consider how we compare to the Nessus Web Application Attack scripts (http://blog.tenablesecurity.com/2009/06/enhanced-web-application-attacks-added-to-nessus.html). [Joao making a list of web scripts which we might find useful, Fyodor asking HD moore for permission to use http enum dir list] o [NSE] HTTP persistant connections/keepalive? May make spidering/grinding/auth cracking more efficient o [NSE] HTTP Pipelining support? May make spidering/grinding/auth cracking more efficient o [NSE] HTTP Cookie suppport? Might be useful for spidering sites which use it for authentication/authorization/personalization. o [NSE] URL grinder checks for existence of applications in common/default paths. Scanning http paths to see if they exist is in some ways similar to scanning to see which ports are open. o Our http-enum does this. o Investigate why and whether we need mswin32/pcap-include/pcap-int.h. This file is not included in the official WinPcap 4.1.1 developers' pack (http://www.winpcap.org/install/bin/WpdPack_4_1_1.zip). Presumably it covers internal functions and structures which we aren't really supposed to access it. If we can get rid of it, that would be great. If we need it, we should probably upgrade to the 4.1.1. version (presumably from the Winpcap source code distribution). Right now it is included in tcpip.h, nsock/src/nsock_pcap.h, and nping/common_modified.cc: o David looked into it. He says it isn't distributed with the WinPcap developer's pack. You have to extract it from the source file. He updated to the 4.1.1 version. He says The entire reason we need it is so we can peek at the definition of struct pcap, so we can access the pcap.adapter member on Windows. In order to pass it to PacketSetReadTimeout. Usually struct pcap is an opaque type and you are only supposed to access it through a pcap_t *. Unfortunately I don't think there's an easy way to manipulate the timeouts in WInPcap like we do on other platforms. You can specify a timeout when you do pcap_open, but we like to set a timeout on every read. So we sort of sneak in and call PacketSetReadTimeout. In the code there's even a comment: "BUGBUG: This is cheating." libdnet also uses the Packet* functions, but in a more innocuous way. It doesn't access them through a struct pcap, so it doesn't need pcap-int.h. David tried testing whether this makes any signficiant difference--to see if we could just remove the PcapSetReadTimeout()--but that didn't work out. - We're not going to worry about this for now since it isn't important enough to pester the pcap people about, and they don't seem to be changing their internal structure anyway. And if they do, we can get the new pcap-int.h. o Further brainstorm and consider implementing more prerule/postrule scripts: o [Implemented] dns-zone-transfer o [Implemented, but a joke] http-california-plates o Investigate this interface-matching problem on Windows: http://seclists.org/nmap-dev/2011/q1/52. It is related to the libdnet changes we made to allow choosing the correct physical interface when teamed interfaces share the same MAC. I think this is solved with the rewritten libdnet code (that uses GetAdaptersAddresses) in my nmap-ipv6 branch. --David o [Ncat] When in connection brokering or chat mode with ssl support enabled, if one client connects and doesn't complete ssl negotiation, it hangs any other connections while that first is active. One way to reproduce: Run SSL chat server like: /usr/local/bin/ncat --ssl -l --chat Window #1: Connect without ssl: ncat -v chatserverip Window #2: Try to connect with SSL: ncat -v --ssl chatserverip Window #2 will not work while #1 is active. If you quit #1, #2 should work again. o IPv6 todo. - Protocol scan (-sO). o [Ncat] Find out what RDP port forwarding apparently doesn't work on Windows. http://seclists.org/nmap-dev/2011/q1/86 o Add raw packet IPv6 support, initially for SYN scan o After that can add UDP scan, and sometime OS detection (David did some research on what IPv6 OS detection might require). o When I (Fyodor) scan scanme.nmap.org with the command "nmap -sC -p80 -Pn -n scanme.nmap.org", I get a blank http-favicon line like: 80/tcp open http |_http-title: Go ahead and ScanMe! |_http-favicon: But if I use "--script http-favicon" instead of -sC, it works fine. o UDP scanning with IP options causes "Received short ICMP packet" on receipt. http://seclists.org/nmap-dev/2011/q1/82 o [Zenmap] Make formerly open ports that are now closed or filtered disappear from the "Ports / Hosts" tab. This appears to be related to ignored states; if in the second scan I use -d2 so all ports are included in the output, the interface is updated correctly. http://seclists.org/nmap-dev/2010/q4/659 o [Zenmap] When a target is unresponsive (and its distance isn't known), put it at the next furthest ring from the known traceroute hosts (with a dashed line), instead of putting it at the first ring. See http://seclists.org/nmap-dev/2011/q1/834. o Rewrite the portreasons code not to use parallel arrays (reason_text, reason_pl_text) and not to require special alignment between the enum codes and (for example) ICMP types. Instead define one structure containing all relevant information about a reason, and define helper functions to map ICMP types to reason codes. In particular, code like this needs to go away: current_reason = ping->type + ER_ICMPTYPE_MOD; if (current_reason == ER_DESTUNREACH) current_reason = ping->code + ER_ICMPCODE_MOD; o Fix memory consumption problem in drda-info (see http://seclists.org/nmap-dev/2011/q2/451) - Fixed (turned out to affect a lot of scripts) o Script dispensation - sip-enum-users and sip-brute. http://seclists.org/nmap-dev/2011/q2/56. o Merged - xmpp. http://seclists.org/nmap-dev/2011/q2/239. o Merged o Script review/disposition: - Merged: DNSSEC enumeration. http://seclists.org/nmap-dev/2011/q1/406. - Merged: quake3-master-getservers patch. http://seclists.org/nmap-dev/2011/q1/925. - Merged: backorifice-info. http://seclists.org/nmap-dev/2011/q2/185. - Merged: omp2-brute and omp2-enum-targets. http://seclists.org/nmap-dev/2011/q2/231. - Merged: http-wp-plugins. http://seclists.org/nmap-dev/2011/q1/806. o Decide what to do about ms-sql-info slowing scans: http://seclists.org/nmap-dev/2011/q1/913 - patch applied: http://seclists.org/nmap-dev/2011/q1/1102 o Script disposition - Patch to get interfaces by Djalal. http://seclists.org/nmap-dev/2011/q1/291 - Incorporated - epmd-info. http://seclists.org/nmap-dev/2011/q1/931. - Incorporated - google-id. http://seclists.org/nmap-dev/2011/q1/952. - Incorporated as http-affiliate-id o [Ndiff] should, in non-verbose mode, perhaps not print the changed Nmap version and/or scan time if nothing else has changed between two files. See http://seclists.org/nmap-dev/2011/q1/674. o Script review disposition: - ssl-known_key http://seclists.org/nmap-dev/2010/q4/733 Thread continues at http://seclists.org/nmap-dev/2011/q1/26. - Merged - dns-nsec-enum - Merged o The file /nmap/mswin32/icon1.ico is used by the NSIS installer to set the Nmap uninstall icon (I'm not sure if it is used for anything else). But this is a very old icon and doesn't match the blue eye we use now. So we should probably update that with a modern "blue insecure eye" icon. I (Fyodor) tried simply replacing icon1.ico with http://insecure.org/shared/images/tiny-eyeicon.ico, but that didn't work. It must not meet the required format. o Add some content to https://secwiki.org and announce it. o Removing -sR option (but keeping the functionality as part of -sV). See http://seclists.org/nmap-dev/2011/q1/688 - Update Nmap documentation/book to remove it there too o Script disposition: - dns-brute by cirrus. http://seclists.org/nmap-dev/2011/q1/351 Should share domain list with http-vhosts. git://code.0x0lab.org/nmap-dns-brute.git - Added by David o Write and post 2010 SoC Successes writeup [Fyodor] o Script review - quake3-master-getservers http://seclists.org/nmap-dev/2011/q1/64 [merged] - dpap-brute by Patrik Karlsson. http://seclists.org/nmap-dev/2011/q1/252. [merged] o The -V option to Nmap, in addition to reporting the version number, should give details on how Nmap was compiled and the environment it is running on. This includes things like whether SSL is enabled, the platform string, versions of libraries it is linked to, and other stuff which is often useful in debugging problems. o We want to list at least: o Nmap version number (that line is fine as is) o host platform string (for which it was compiled) o Whether OpenSSL and LibSSL, NLS, and IPv6 are enabled - Version number of OpenSSL and LibSSL if those are enabled o Version numbers of libdnet, libpcre, and libpcap o Script review: - SCADA scripts http://seclists.org/nmap-dev/2010/q4/612 http://seclists.org/nmap-dev/2010/q4/613 http://seclists.org/nmap-dev/2010/q4/623 http://seclists.org/nmap-dev/2010/q4/639 [on hold] - servicetags http://seclists.org/nmap-dev/2010/q4/691 needs new testing on OpenSolaris: http://seclists.org/nmap-dev/2011/q1/91 [committed] - firewalk-path http://seclists.org/nmap-dev/2011/q1/63 [committed over previous firewalk script] - snmp-ios-config http://seclists.org/nmap-dev/2011/q1/10 Requires a TFTP server; decision was to build such server in Lua if possible. Patrik Karlsson's beginning TFTP implementation: http://seclists.org/nmap-dev/2011/q1/169. [committed by Patrik] o Script merged: p2p-dropbox-listener http://seclists.org/nmap-dev/2010/q4/689 o A trivial change: we currently print some lines about NSE pre-scanning and post-scanning in verbose mode even when no such scripts are being run. We should not print those in that case. For example, nmap -A -v scanme.nmap.org gives me these superfluous lines: NSE: Script Pre-scanning. NSE: Starting runlevel 1 (of 2) scan. Initiating NSE at 12:23 Completed NSE at 12:23, 0.00s elapsed NSE: Starting runlevel 2 (of 2) scan. NSE: Script scanning 64.13.134.52. NSE: Starting runlevel 1 (of 2) scan. Initiating NSE at 12:24 Completed NSE at 12:24, 4.14s elapsed NSE: Starting runlevel 2 (of 2) scan. NSE: Script Post-scanning. NSE: Starting runlevel 1 (of 2) scan. NSE: Starting runlevel 2 (of 2) scan. o Do new Nmap release with the stuff merged from SoC students and other new developments. o Modify Zenmap to use the new --script-help system to enumerate scripts and collect information such as their descriptions. This will resolve the problem of Nmap's broadcast prerule scripts running when you open the profile editor. o Document --script-help in docs/refguide.xml and docs/scripting.xml. o [Zenmap] Brian Krebs found a problem (which Fyodor is able to reproduce) in the target selector on the left pane. When you select one of the scanned targets, it is supposed to jump to that target in the "Nmap Output" tab on the right pane. Instead, nothing seems to happen. One of our output format changes probably broke the feature. It still works fine if you have the "Ports / Hosts" or "Host Details" tabs active in the right pane instead. o Include a --script-help system to Nmap, which provides user readable text help and also machine parsable XML information for scripts which match a pattern (e.g. the same sort of arguments you could use for --script, like a category or http-* or whatever). The --script-help ONLY provides help and quits, it does not run the script. For some initial implementation work, see this thread: http://seclists.org/nmap-dev/2011/q1/163 o [Nping] See whether --echo-client mode really requires root, and remove that restriction if not. Luis explanation for requiring root: http://seclists.org/nmap-dev/2011/q1/248 o Script review: - p2p-dropbox-listener http://seclists.org/nmap-dev/2010/q4/689 o Decide whether to include NSE console script help, decide on implementation issues. http://seclists.org/nmap-dev/2011/q1/163 o [Zenmap] Use a more efficient algorithm to update the display of Nmap normal output in live scans. zenmapGUI.NmapOutputViewer.NmapOutputViewer.refresh_output calls zenmapCore.NmapCommand.NmapCommand.get_output, which re-reads the entire output file (into memory) and then puts it in the text buffer if it has changed. So already we're storing the whole output twice in memory. When the text field changes, update_output_colors re-highlights the whole file. o Update changelog to note recent changes o Do final dev/test release o If Nping is compiled w/o SSL support, and the user specifies an encryption key, it should fail and insist they use --no-crypto rather than ignoring the key and omitting crypto. Otherwise the user might think they're getting encryption when they're not. David found this problem in the server, and we also should check how the client behaves. o [Ncat] Make --exec work in conjunction with --proxy. The --proxy code path skips the --exec code. See http://seclists.org/nmap-dev/2010/q4/604 and the test "--exec through proxy" in ncat-test.pl. o Decide what to do about Nmap static binaries failing to work on new Fedora releases (and others?). See these threads: http://seclists.org/nmap-dev/2011/q1/46 and http://seclists.org/nmap-dev/2010/q1/308 o We ended up dynamically linking system libs in the RPM rather than statically linking them. We still statically link things like lua, pcre, ssl, etc. o Fix our mac builds so that they contain SSL support again (5.35DC1 did, but TEST1 and TEST2 didn't for some reason. o Add our broadcast discovery scripts to a "broadcast" category (they should generally just be in "broadcast" and (assuming they are safe) "safe", and not normal "discovery". Update scripting.xml to note this new category too. o The latest IANA services file (http://www.iana.org/assignments/port-numbers) has many identified services which are still "unknown" in our files because ours is based on a much older version of that file. We should probably take that file and add names and comments to our nmap-services-all where they are "unknown" in our file. An example of such a port is 3872, oem-agent. o Script review: - patch for ftp-proftpd-backdoor http://seclists.org/nmap-dev/2010/q4/678 - patch for hddtemp-info http://seclists.org/nmap-dev/2010/q4/676 o We should probably update our Windows build systems to use Python 2.7. As of 11/8, it looks like all our dependency libraries are available for 2.7: o David upgraded and it worked, though Rob found a potential problem and added vcredist 2008. Fyodor will test on the official Win7 Nmap build system. PyGTK: 2.22.0 IS available for 2.7 PyCairo: 1.8.10 IS available for 2.7 PyGObject: 2.26.0 IS available for 2.7 Py2exe: 0.6.9 IS available for 2.7 o Do service/version detection submission integration (last done in April) o Do os detection submission integration (last done in April) o Script review: - modbus-enum http://seclists.org/nmap-dev/2010/q4/489 o Create Nmap wiki o Decide on domain name o Include insecure Chrome o Decide on wiki software, probably just use mediawiki o install it on a Linode, probably Web o [NSE] Web application fingerprinting script. Would be great to be able to take a URL and determine things like "this is Joomla" or "this is Plone" or "Mediawiki" or whatever. Rather than hard code regular expressions or other tests in a script, it should use a signature file like Nmap OS and version detection do. Might work in combination with URL grinder to check for applications at default/common locations. See also a script that does favicon scanning TODO item. - http-enum pretty much does this now. o Update our distribution build systems and documentation to use Visual C++ 2010 Express rather than the 2008 version. See http://www.microsoft.com/express/Windows/ o Dependency licensing issues (OpenSSL, Python, GTK+, etc.) o Almost done! We just have some file renaming/organizing left to do. o We should do an audit to ensure that we are in complete compliance for the licenses of all the software we ship in any of our downloads, as some licenses have special clauses for things like including their license/copyright file, mentioning them in our documentation, etc. And of course we want to credit them properly even where the license doesn't require it. We should probably make a list of these in our docs/ directory along with any special information/requirements of their license. And maybe we should put the current licenses in a subdir too. In particular, these come to mind: o libpcre o lua o OpenSSL o libpcap o GTK+/Glib/ATK/Pango/PyGTK (Win/Mac versions of Zenmap link to PyGTK) o SQLite o Python (Win/Mac versions of Zenmap link to Python) o X.org libraries (Mac version links to them) o libdnet o Small NSEDoc bug: https://nmap.org/nsedoc/scripts/dns-zone-transfer.html contains 'id \222\173' near the bottom. This is presumably due to misparsing this line from the script: local req_id = '\222\173'. Given that we don't use IDs any more, maybe we can just get rid of the functionality. o [NSE] We should probably enable broadcast scripts to work better by (initial thoughts): o Done and merged by David! 1) Change NSE to always set nsp_setbroadcast() on new sockets 2) Change nsock to create real sockets at time of nsi_new so you can bind to them. See this thread (only some of the messages involve broadcast support): http://seclists.org/nmap-dev/2010/q3/357 o [NSE] Review scripts: o Hostmap (Ange Gutek) - http://seclists.org/nmap-dev/2010/q3/159 o Post BH/Defcon Nmap videos o Let Nsock log to stderr, so its messages don't get mixed up with the output stream when Ncat is run with -vvv. http://seclists.org/nmap-dev/2010/q3/113 o [NSE] Our http-brute should probably support form POST method rather than just GET because some forms require that. o Nping needs to call nsp_delete so that its socket descriptors are not left behind. o [Zenmap] Add a button to select script files from the filesystem. o [Zenmap] Show help for individual script arguments in the Help pane, not for all arguments at once. o Upgrade our Windows OpenSSL binaries from version 0.9.8j to the newest version (1.0.0a as of Aug 12, 2010). o Since Libdnet files (such as ltmain.sh) are apparently only used by libdnet (they used to be used by shared library NSE C scripts), we should move them to the libdnet directory. o Turned out to be a pain. See http://seclists.org/nmap-dev/2010/q3/733 o [Zenmap] Consider a memory usage audit. This thread includes a claim that a 4,094 host scan can take up 800MB+ of memory in Zenmap: http://seclists.org/nmap-dev/2010/q1/1127 The reporter mentioned Guppy/Heapy to debug memory use: http://guppy-pe.sourceforge.net/ http://www.pkgcore.org/trac/pkgcore/doc/dev-notes/heapy.rst. Many Nmap survey respondants complained about this too. Note: Fyodor has a 50MB scan log file named ms-vscan.xml which demonstrates this problem. When trying to load the file, Zenmap grows to 1150MB of RAM, pegs the CPU usage at 100% for many minutes or maybe hours (I forgot about it, but woke up the next day to find that it had started, was then using 2.4GB of RAM. The hosts/services functionality seemed to work, although it would take a minute or so to switch from say "ftp" port to view "ssh" ports. o [NSE] Maybe we should create a script which checks once a day whether similar tools (Metasploit, Nessus, OpenVAS, etc.) have any new modules, and then mails out a list of them with the description fields. The mail could go to just interested parties, or maybe nmap-dev. This may help prevent important vulnerabilities from falling through the cracks. Perhaps we would include new NSEs in there too, especially if we open it up as a public list. o Now that NSE has more script phases (prerule, postrule, hostrule, portrule, and versionrule soon to come), the NSEDoc should specify which phases a script belongs to. o Consider implementing a nsock_pcap_close() function or making nsp_delete() call pcap_close() when pcap IODs are used. Currently valgrind warns about a socket descriptor left opened (at least in Nping). See http://seclists.org/nmap-dev/2010/q3/305. o It turns out that the pcap descriptors are being closed properly, but Nping isn't calling nsp_delete. o [NSE] High speed brute force HTTP authentication. Possibly POST and GET/HEAD brute force cracking. [done except for form POST, adding separate TODO item for that] o [NSE] Review scripts: o New brute, vnc, and svn scripts by Patrik. This guy is a coding machine :). http://seclists.org/nmap-dev/2010/q3/111 o rmi-dumpregistry by Martin Swende. http://seclists.org/nmap-dev/2010/q2/904 o path-mtu.nse - http://seclists.org/nmap-dev/2010/q3/222 o 15 more from Patrik :). http://seclists.org/nmap-dev/2010/q3/284 o [NSE] Consider modifying our brute force scripts to take advantage of the new NSE multiple-thread parallelism features. - We've done this with db2-brute, but the DB may have been a bottleneck there, so we should probably do more testing after modifying another script for this sort of parallel cracking. o Look into implementing security technologies such as DEP and ASLR on Windows: http://seclists.org/nmap-dev/2010/q3/12. o Ncat and Nmap should probably support SSL Server Name Indication (SNI). See this thread: http://seclists.org/nmap-dev/2010/q3/112. We need this to talk to web servers which share one SSL IP and port because we need to ask for the right SSL key. o [NSE] In the same way as our -brute scripts limit their runtime by default, I think qscan should be less intense by default. For example, perhaps it could run by default on no more than 8 open ports, plus up to 1 closed port. Right now it does things like running on 65,000+ closed ports and bloats scan time (and output). Of course there could (probably should) still be options to enable more intense qscanning. o [Web] We should see if we can easily put the Insecure chrome around Apache directory listings and 404 pages (e.g. https://nmap.org/dist/ and https://nmap.org/404). I think we may have had this working before the move to Linode, so maybe check conf/httpd.conf.syn. o Do a serious analysis if and how we should use the NIST CPE standard (http://cpe.mitre.org/) for OS detection and (maybe in a different phase) version detection results. One thing to note is that they may not have entries for many vendors we have. For example, one person told me they couldn't find SonicWall or D-Link in the CPE dictionary. Here are some discussions threads on adding CPE to Nmap: http://seclists.org/nmap-dev/2008/q4/627 and http://seclists.org/nmap-dev/2010/q2/788. Nessus has described their integration of CPE at http://blog.tenablesecurity.com/2010/05/common-platform-enumeration-cpe-with-nessus.html. o [NSE] Create NSE scripts to scan for and/or exploit these VXWorks issues: http://blog.metasploit.com/2010/08/vxworks-vulnerabilities.html [Ron may be able to do this. Or others are welcome to take a shot at it.] o The -g (set source port) option doesn't seem to be working (at least in Fyodor's quick tests) for version detection or connect() scan, and apparently doesn't work for NSE either. We should fix this where we can, and document the limitation in the refguide where it is impractical. Also see http://seclists.org/nmap-dev/2010/q2/576. o [Zenmap] script selection interface for deciding which NSE scripts to run. Ideally it would have a great, intuitive UI, the smarts to know the scripts/categories available, display NSEdoc info, and even know what arguments each can take. o Review http-xst (Eduardo Garcia Melia) - http://seclists.org/nmap-dev/2010/q3/159 o [NSE] Investigate sslv2.nse falsely reporting SSLv2 as being supported. http://seclists.org/nmap-dev/2010/q2/754 o [NSE] The NSEDoc for some scripts includes large "Functions" sections which aren't really useful to script users. For example, see https://nmap.org/nsedoc/scripts/snmp-interfaces.html. Perhaps we should hide these behind an expander like "Developer documentation (show)". I don't think we need to do this for libraries, since developers are the primary audience for those documents. o Talked to David. We should just remove the function entries. o We should add a shortport.http or similar function because numerous services use this protocol and many of our scripts already try to detect http in their portrule in inconsistent ways. o [NSE] Maybe we should create a class of scripts which only run one time per scan, similar to auxiliary modules in Metasploit. We already have script classes which run once per port and once per host. For example, the once-per-scan ("network script"?) class might be useful for broadcast LAN scripts (Ron Bowes, who suggested this (http://seclists.org/nmap-dev/2010/q1/883) offered to write a NetBIOS and DHCP broadcast script). Another idea would be an AS to IP ranges script, as discussed in this thread http://seclists.org/nmap-dev/2010/q2/101 [Could be a good SoC infrastructure project] o David notes: "I regret saying this before I say it, because I'm imagining implementation difficulties, we should think about having such auxiliary scripts be able to do things like host discovery, and then let the following phases work on the list it discovers." o Analyze what sort of work would likely be required for Nmap to support OS detection over IPv6 to a target. o Would probably start with a way to send raw IPv6 packets o There is a raw IPv6 patch here: http://seclists.org/nmap-dev/2008/q1/458 o Also it looks like Nping may be doing this already. o Then we need to figure out if we can use our current DB and techniques, or if we'd likely thave to have an IPv6-specific DB. [David] o July Nmap releases (at least a beta version, and maybe a stable too). Last release was 5.30BETA1 on March 29 o Add this patch for compilation on OpenSolaris. http://blogs.sun.com/sdaven/entry/nmap_5_35dc1_compile_on o Now that we've put the ndiff, ncat, and nping man pages under the scope of the book (e.g. https://nmap.org/book/ncat-man.html), we need to add a redirect from the old locations and also update our links. o Make sure the long output lines in Nping's man page are OK for the book. See r18829 and r18864. o Update "History and Future of Nmap" (https://nmap.org/book/history-future.html) to include all the news since September 2008. [Fyodor] o Fix Win7 networking issue reported by Luis which seems to have been triggered by r17542. See this thread: http://seclists.org/nmap-dev/2010/q3/40 o Upgrade to WinPcap 4.1.2 - Rob has a patch - See this thread: http://seclists.org/nmap-dev/2010/q3/18 o [NSE] Review UnrealIRCd backdoor detection script http://seclists.org/nmap-dev/2010/q2/854 o [Zenmap] Investigate segfault on some installs of OS X 10.6.3: http://seclists.org/nmap-dev/2010/q2/587 o David rebuilt with MacPorts 1.9.1 rather than 1.8.2 and the problem went away. o [Zenmap] Investigate failure to start on some installations of OS X 10.6.3. [ We think one may just not have waited long enough as he said it started working, and another case (the 587) seems to be a segfault--we added a new task for that ] http://seclists.org/nmap-dev/2010/q2/587 http://seclists.org/nmap-dev/2010/q2/859 (He responded to David privately and said that it was not an I7 processor.) Nmap seems to be having problems too: http://seclists.org/nmap-dev/2010/q2/747 o [NSE] Review Gutek's PHP version disclosure script. http://seclists.org/nmap-dev/2010/q2/569 o Fix the IPv6 name resolution problem described in this thread: http://seclists.org/nmap-dev/2010/q2/787 o [NSE] Review Gutek's libopie detection/DOS script. http://seclists.org/nmap-dev/2010/q2/635 o [NSE] Review Gutek's web server directory traversal script. http://seclists.org/nmap-dev/2010/q2/595 - It became modifications to http-passwd o [NSE] Review dns-cache-snoop.nse from Eugene Alexeev. http://seclists.org/nmap-dev/2010/q2/195 Better attachment at: http://seclists.org/nmap-dev/2010/q2/200 Need to decide on a domain list: http://seclists.org/nmap-dev/2010/q2/199 o Fix bug where multiple targets with the same IP can end up in a hostgroup and cause port scanning and probably OS detection to misbehave. An example is "nmap -F scanme2.nmap.org scanme3.nmap.org". See this thread for details: http://seclists.org/nmap-dev/2010/q2/322 o Need to fix our current win32.zip distribution so that .svn files aren't included (currently they are in nselib/data). Will probably be a simple adjustment to mswin32/Makefile. o Make Zenmap splash screen o [NSE] Add one of, or combine, ntp-peers and ntp-monlist. http://seclists.org/nmap-dev/2010/q2/190 http://seclists.org/nmap-dev/2010/q2/191 o [NSE] Reorganize nselib to allow libraries in subdirectories. Currently, to avoid expanding the number top-level libraries, code that is only used by one library is built into that library's file, even if it is logically separate. For example, the mongodb library contains a BSON-parsing library. Instead, that library could go in mongodb/bson.lua. The msrpc and smb libraries could potentially be broken up in this way. UPDATE: We decided not to do this for now, given complications in nsedoc, packaging, etc. to support the new hierarchy. Instead, we can use prefixes like we do with scripts (e.g. mongodb-bson.lua, msrpc-types.lua). o Add a configure option to our libpcap which enables an older Linux packet capture system (David's noring patch). This is needed in some cases for 32-bit static binaries to work on 64-bit Linux systems. Note that it is unneccessary if both the build system and the target system use Linux 2.6.27, as that has an architecture independent tpacket_hdr (called tpacket2_hdr). [Added by David as --disable-packet-ring] o Test Jay Fink's UDP payload prototype. http://seclists.org/nmap-dev/2010/q1/168 [ tested, improved, merged by David] o Resolve Ncat broadcast support issue (see this thread: http://seclists.org/nmap-dev/2010/q2/422). o [NSE] Review and test the DB2 library and scripts. http://seclists.org/nmap-dev/2010/q2/395 (but updated versions may be available). o Move nmap/docs/TODO into its own todo directory (probably nmap/todo) and then encourage maintainers of /status/ TODOs and any other TODOs to migrate theirs there. Unlike the status directory, /nmap/todo would be readible by anyone. [Fyodor] o Nmap should at least print (and maybe scan) all IP addresses for hostnames specified on the command line. We will start with just printing all the addresses. Here is a thread on the topic: http://seclists.org/nmap-dev/2010/q2/302 [David made it do the printing, adding a different task related to scanning them all] o Integrate new service detection fingerprint submissions (we have more than 730 since Dec. 17, 2009. o [Ncrack] Use our new password lists (now used by NSE) for Ncrack as well. Ncrack can probably handle a larger list than NSE uses. o Consider MSRPC ideas from Ron--we might want to add some as TODO tasks: http://seclists.org/nmap-dev/2010/q2/389 o Fix XML inconsistency described at http://seclists.org/nmap-dev/2010/q2/326 o Integrate new OS fingerprints (we have more than 1,300 since November 10, 2009). o Finish selecting GSoC 2010 projects o Upgrade libpcap to the new 1.1.1 version. o Improve the NSI installer by adding command-line options for unsetting each of these GUI checkboxes individually (particularly useful for silent mode): LangString DESC_SecCore ${LANG_ENGLISH} "Installs Nmap executable, NSE scripts and Visual C++ 2008 runtime components" LangString DESC_SecRegisterPath ${LANG_ENGLISH} "Registers Nmap path to System path so you can execute it from any directory" LangString DESC_SecWinPcap ${LANG_ENGLISH} "Installs WinPcap 4.1 (required for most Nmap scans unless it is already installed)" LangString DESC_SecPerfRegistryMods ${LANG_ENGLISH} "Modifies Windows registry values to improve TCP connect scan performance. Recommended." LangString DESC_SecZenmap ${LANG_ENGLISH} "Installs Zenmap, the official Nmap graphical user interface. Recommended." LangString DESC_SecNcat ${LANG_ENGLISH} "Installs Ncat, Nmap's Netcat replacement." LangString DESC_SecNdiff ${LANG_ENGLISH} "Installs Ndiff, a tool for comparing Nmap XML files." LangString DESC_SecNping ${LANG_ENGLISH} "Installs Nping, a packet generation tool." o We should have a standard function which takes time arguments in the same format as Nmap does (e.g. 60s, 1m, etc.) and the scripts which take time arguments should be modified to use it. David suggests this here: http://seclists.org/nmap-dev/2010/q2/35. We are also going to update the normal Nmap timing functions to take seconds by default, as described here: http://seclists.org/nmap-dev/2010/q2/159 o Nmap should probably always produce a well-formed XML file, even if it exits with a fatal() error. In that case, the error should be included in the XML. Right now, for example, if the network is down, the XML output will just stop (no closing tags) and Nmap will print something to STDERR like: nexthost: failed to determine route to 9.48.184.164 QUITTING! o Get @output sections for the last remaining scripts w/o them: [WARN] script auth-spoof missing @output [WARN] script db2-das-info missing @output [WARN] script db2-info missing @output [WARN] script http-passwd missing @output [WARN] script iax2-version missing @output [WARN] script ms-sql-config missing @output [WARN] script ms-sql-query missing @output [WARN] script oracle-sid-brute missing @output [WARN] script pop3-brute missing @output [WARN] script pptp-version missing @output [WARN] script skypev2-version missing @output o [Zenmap] Maybe it should sort IPs in an octet-aware way. And maybe you should be able to sort by IP address (perhaps that should be the default). Current plan is to just sort by IP by default, and maybe we'll offer other sort techniques later if desired. See http://seclists.org/nmap-dev/2010/q2/27 [possible SoC student task] o Brainstorm for GSoC 2010 ideas and fill out the org application by Friday 3/12 4PM PST. o NSE scripts o Maybe a whole SoC role for http scripts o Maybe look at other web app scanners for some inspiration (including w3af - http://w3af.sourceforge.net/) o Maybe a non-http developer too o NSE infrastructure manager o Ncrack o Nping o Mobile Devices? N900, iPhone, Android o Zenmap developer o Must have solid user interface design experience o Zenmap script selector (subset of a Zenmap or NSE SoC role) o Feature Creepers/Bug fixers o Review IDS detection scripts from Joao Correa. http://seclists.org/nmap-dev/2010/q1/814 o Review mssql library and scripts from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/1000 (files) http://seclists.org/nmap-dev/2010/q1/1014 (sample output) o Review DNS fuzzer script from Michael Pattrick. http://seclists.org/nmap-dev/2010/q1/1005 o Our nsedoc generator should probably give a warning if a script is missing any important fields. @output comes to mind. @usage can be nice too, though we could consider auto-generating that for trivial scripts. o [NSE] Consider pros and cons of splitting information retrieval scripts into a bunch of small single-purpose script vs. one larger argument-controlled script. See http://seclists.org/nmap-dev/2010/q1/1023 [we ended up combining three of the ms-sql scripts. If we combine future scripts, we need to remember to add them to the deprecation list in the Makefile] o Remove --interactive. It was broken for a long time and nobody seemed to notice, and we put a call out on nmap-dev for --interactive users and didn't get any good reasons to keep it. We should kill it to remove the code complexity it adds and to avoid the documentation complexity of people having to read and learn about a feature they are unlikely to ever use. o Zenmanp should perhaps be able to print Nmap output on a Printer (if not too much of a pain to implement.) o Review afp-serverinfo.nse from Andrew Orr. http://seclists.org/nmap-dev/2010/q1/470 Just waiting on some bug fixes: http://seclists.org/nmap-dev/2010/q1/665 o Test 64-bit pcap installer (e.g. remove old version and install new) before next release, as we've applied a change from Rob which works on his system (http://seclists.org/nmap-dev/2010/q1/796). o [NSE] Improve username/password library (the database files themselves). We don't have very good lists at the moment. Maybe work in combination with Ncrack dev. o Now there are some even better lists available (f.e. RockYou)--see this thread: http://seclists.org/nmap-dev/2010/q1/764 o We've improved the ncrack files--we should probably either use those for NSE or use a subset of them. o perhaps from Solar Designer. (he sent us permission) o perhaps add phpbb hack data (there is at least a list of 28,635 passwords in phpbb_users.sql, and possibly more in other files. o [Nping] Should take the version number 0.[nmap version], such as 0.5.22TEST o Review rpc.lua, nfs-showmount.nse, nfs-get-stats.nse, and nfs-get-dirlist.nse from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/270 o [NSE] Look into moving packet module to C for better performance [Patrick] o Removing this one because it is stale (has been here for many months with no action seen), but it is something we can consider if/when there is a desire to implement it. A key is probably to measure current performance and see if it is a material problem. o Maybe the Nmap ASCII art should come after make rather than configure? - We decided it would probably be annoying for developers to see it every time they 'make'. o Review snmpenum.nse from William Njuguna. http://seclists.org/nmap-dev/2009/q4/721 http://seclists.org/nmap-dev/2010/q1/656 o Dropping for now unless original author or someone else picks it up and fixes the bugs. o Add smtp-enum-users from Duarte Silva if testing is favorable. http://seclists.org/nmap-dev/2010/q1/699 o After the new -sn and -Pn options (added to SVN around 7/20, just after the 5.00 release) have been around long enough to be in most people's copy of Nmap (e.g. in all the versions we distribute from download page (stable+dev)) for at least a few months, we'll document these as the preferred version rather than -sP and -PN. These match -n, and the main problem with -sP is that we now use it more for "disable portscan" than ping only. For example, you can also use NSE, traceroute, etc. [David] o Nmap currently selects routes based on the first matching one it finds. But it should really take the most specific route instead. So it should: 1) Keep searching the routing table for the most specific match, and 2) Use a stable sort (not qsort) so that routes with identical netmasks aren't rearranged. For more, see http://seclists.org/nmap-dev/2010/q1/685 o Review pgsql-brute.nse from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/455 o psexec missing (need to download yourself now) nmap_services.exe output issue: "The function where this is detected returns a value that is passed to stdnse.format_output. format_output takes a parameter to decide whether it's displaying an error message, but it is hard-coded to only display error messages with debugging >= 1. So options are to change format_output and make it more flexible, or somehow decouple the sensing of nmap_service.exe from the normal output channel of the script." o Website: Create shared directory in svn, which will contain directories shared between the Insecure.org network of sites (e.g. templates, error, css). Then sites such as sectools, nmap.org, insecure.org can just check that out via externals declaration (or, I suppose, symlink). CSS directives will then use /shared/css/insecdb.css etc. ). o Add CouchDB and JSON scripts once the JSON library is finished. http://seclists.org/nmap-dev/2010/q1/641 o Review NSE raw IP from Kris Katterjohn. http://seclists.org/nmap-dev/2010/q1/559 o Review sslv3-enum.nse from Mak Kolybabi. http://seclists.org/nmap-dev/2010/q1/563 o [NSE] Consider LDAP library and scripts from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/70 [all merged, except David is still reviewing ldap-search] o More potential improvements to http-methods: http://seclists.org/nmap-dev/2010/q1/630 and http://seclists.org/nmap-dev/2010/q1/640 o Remove smtp-open-relay.nse sometime after 9/24/09 if nobody adopts it (see http://seclists.org/nmap-dev/2009/q3/0986.html). [It got fixed up and we kept it.] o The -v and -d arguments should take the same syntax. Right now you use -vvv vs. -d3. We should probably just make either approach work with either of them. o Zenmap should be able to export normal Nmap output o Integrate Nping. o [NSE] Consider the http-methods script from Bernd Stroessenreuther. http://seclists.org/nmap-dev/2010/q1/76. [integrated, but David is making some improvements]. o The Nmap web page is beginning to show its age. Ah, who am I kidding, it was showing its age 5 years ago :). It could do with an upgrade to XHTML+CSS. It could also do with a whole redesign, but I think that can be done as a second step after converting to XHTML+CSS with roughly the same look. Though adding a few more modern touches (like hover interaction on the menu bar) wouldn't hurt. This is a moderatly big project, which will involve: o Designing the new XHTML+CSS to look similar to the current HTML pages, but be extensible enough that it can be redesigned in the (near) future by mostly just changing the CSS and graphics. o Converting the existing Nmap pages to the new XHTML format. This will likely include using open source programs and likely modifying them or creating your own scripts to help with the process. To apply for this task, you need to have some web development experience and an example XHTML+CSS web page you have created online. o We decided not to worry about XHTML for now, and we're integrating CSS in piece by piece -- we already have the section headers, left sidebar links. etc. o Should not use SSI like the current pages -- should do all its magic through CSS. That way it will work on seclists too (which can't do SSI for security reasons). o Maybe alpha transparency for menus, gradiants, curves, etc. But the main goal isn't flashiness. o Seclists.org should maybe be fixed so that it doesn't strip quoted text for its summaries from the IP list because that list consists almost entirely of forwarded material which is being stripped. Look at the summaries at http://seclists.org/interesting-people/. o Web site HTML improvements - Maybe start with nmap.org. - Find and fix HTML validation problems, bad links. I'm not sure what tool is best for this. - Then do the same with seclists.org, insecure.org, sectools.org - The icon on the top-left of the screen should be for (and link to) the root URL of current site. e.g. seclists.org, sectools.org, nmap.org rather than always insecure.org. o [NSE] Consider SNMP scripts from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/162 http://seclists.org/nmap-dev/2010/q1/174 http://seclists.org/nmap-dev/2010/q1/178 o Deal with AV false positive issue RE nmap_services.exe: - For now, David is going to apply Ron's patch which removes this, but David will make it print output in verbose mode rather than debug and maybe make it a little less verbose. LT plan is for Ron to encrypt it with OpenSSL. o Web site improvements - Update to use CSS, at least for header bars - Also, if it is easy to give the header bars rounded corners, we should probably do so. But if it is hard, it isn't important enough to matter. - The Nmap.Org navigation table should have a background and more subtle lines, like we use for our calendars now. - The first item (table) in featured news has slightly more left/right margin than the later ones on Firefox 3.5.6, and with IE8 it doesn't extend as far when you make the page really wide. Plus the images on the right are problematic (extend through the border below them) when you make the window too wide on IE8. Having a slight margin on the left/right of entries would actually be a bit nice. And it would be nice if it only took a simple tag or two, controlled by CSS rather than pasting in a whole table with font tags and the like for each entry. o [Ncat] Test, review, and (if appropriate) merge Venkat's HTTP Digest proxy authentication patch. See http://seclists.org/nmap-dev/2009/q3/773. [David] o [NSE] Look at new DB2 script by Tom Sellers. http://seclists.org/nmap-dev/2009/q4/659 o [NSE] Consider MongoDB scripts and libraries from Martin Holst Swende. http://seclists.org/nmap-dev/2010/q1/177 o [NSE] Document Patrick's worker thread patch in scripting.xml (see http://seclists.org/nmap-dev/2009/q4/294, https://nmap.org/nsedoc/lib/stdnse.html#new_thread, https://nmap.org/nsedoc/lib/nmap.html#condvar) [Patrick] o Make Nmap 5.21 bugfix-only release o [NSE] Consider afp-showmount script from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/97 [merged to trunk] o [NSE] Review DNS-SD script from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/87 [merged to trunk] o [NSE] Consider MySQL scripts from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/163 [merged to trunk] o [NSE] Consider DAAP script from Patrik Karlsson. http://seclists.org/nmap-dev/2010/q1/164 [merged to trunk] o NSEDoc left sidebar should include a link to https://nmap.org/book/nse.html below "Index". o Consider enhancing the new OS Assist system to handle version detection too. [We decided not to do this as David noted that Doug's serviceunwrap.lisp does pretty much everything he needs.] o [NSE] HTTP header parsing is not very robust, and is duplicated in a lot of places. For example, it's legal to have header fields like Content-type:\r\n ___text/html\r\n (with spaces in place of _, but http.lua won't parse such a header correctly. In other words you can extend them to any number of lines as long as each line after the first begins with whitespace. [David] o Investigate issue with our Pcap and Wireshark x64, as described in this thread: http://seclists.org/nmap-dev/2009/q4/557 [Rob] [Taking this off the list until/unless we get more reports] o Decide what to do about Windows 7/Vista and starting NPF. See this thread: http://seclists.org/nmap-dev/2010/q1/20 o [NSE] We should do a favicon survey like the one Brandon did for /favicon.ico files but which uses the favicons specified by the HTML files rather than just that exact location. For example, insecure.org sites include in the headers: Then we should update our favicon database to include the top ones, and we should also improve our favicon script so that it either omits checking /favicon.ico if the HTML-specified one exists, or it should just download, interpret, and display info for both (right now it seems to give prority to the wrong one: /favicon.ico). o [Ncat] Add SSL support for --exec so you can use SSL to talk to your remote shell, etc. See this thread: http://seclists.org/nmap-dev/2009/q4/255, particularly the implementation sketch at http://seclists.org/nmap-dev/2009/q4/268 [Venkat,David] o Look at new Kerberos script from Patrik Karlsson. http://seclists.org/nmap-dev/2009/q4/715 . [We decided not to merge this one since its usefulness turned out to be limited on Windows and very limited on any other platform. ] o Add feature to http library to let user set the user agent to be used. The NSEDoc for this feature should probably tell what our current default user agent is ("Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html") [David] o On our NSEDoc pages (e.g. https://nmap.org/nsedoc/), perhaps the link text for scripts should not include the ".nse". Basides saving horizontal space, this may improve the sorting so that the likes of "citrix-enum-apps" comes before "citrix-enum-apps-xml". Also, we can probably get away with reducing the width of the NSEDoc left-column, especially if ".nse" is removed. o [NSE] Patrick's script dependency patch: http://seclists.org/nmap-dev/2009/q4/295 o I'm not sure if he has gone through and actually set appropriate dependencies (and removed runlevels) yet o Integrate latest version detection submissions and corrections. This was last done based on submissions until February 9, 2009. o Release 5.10BETA2 o Add --evil to set the RFC3514 evil bit. ftp://ftp.rfc-editor.org/in-notes/rfc3514.txt o We're not going to add this right now. o Talk to Libpcap folks about incorporating (at least some of) my changes from libpcap/NMAP_MODIFICATIONS. [marking as done since the upstream-appropriate changes are pretty minor now that we've upgraded to 1.0] o Nping -- like hping3 but uses Nmap infrastructure and to a large degree the same command-line options as Nmap. [We now have an alpha version at https://nmap.org/nping/] o Further investigate SCTP functionality, as some people reported problems (see this thread: http://seclists.org/nmap-dev/2009/q2/0669.html) o [NSE] NFS query script for checking exports, etc.? [Patrik Karlsson] o [NSE] Attempt to reproduce and fix a deadlock reported by Brandon when he does large-scale scanning with a new favicon script with hostgroups as small as 8,192 (he hasn't seen it with 4096 hostgroups). Could be a bug in internal NSE socket lock. Probably not specific to the favicon script, but that is how Brandon reproduces it. At the hang, stack trace is usually the threads stuck in socket_lock function, sometimes lookup_cache mutex in http library. David guesses that it's threads being garbage-collected from the socket lock table. The only thing that can wake up a thread waiting on a socket lock is if a thread that holds a lock is removed from the table. But the table has weak keys, meaning that a thread can be garbage collected and it will be automatically removed from the table by the Lua runtime. Then there is no event that can wake up a thread waiting for a lock. [David and Patrick made some commits at end of November meant to resolve this, and we haven't seen the problem since, so we're marking it as done for now]. o Look into reducing Nmap memory consumption o UDP scans with -p- and large hostgroups are a particularly large offender. See if there is a way to prevent them from eating up gigs of RAM. See the message "Port memory bloat" at http://seclists.org/nmap-dev/2009/q3/0926.html for a patch that reduces Port memory use by about 50%. o One idea David has been considering is a way to represent filtered ports (or whatever the default state is) without creating a Port object for each one. [David] o Fix assertion failure with certain --exclude arguments (see http://seclists.org/nmap-dev/2009/q4/276). [David] o Many people may have stale (since removed/renamed) scripts in their Nmap scripts directory because our 'make install' does not remove them and so they remain and can cause problems (like running twice after being renamed). We should probably add a line to our 'make install' which removes the scripts/lib names we have previously used. We're doing this rather than blowing away the old directory just in case someone has custom scripts/libs there (though that is still a bad idea). [David] o Update the CHANGELOG for new 5.10BETA1 release. [Fyodor] o Make the new Nmap 5.10BETA1 release o Ndiff man page should be built from XML source whenever a release is done, as ncat/zenmap/nmap man pages are. [Fyodor] o We should package the rendered Nroff man page translations (e.g. all 16 languages) in the tarball to make it easier for distributors to package them. For example, see http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=358336. Including the translations would add 2.5MB to the (currently 28MB) uncompressed tarball and about 800KB to the (currently 9MB) bz2 compressed tarball. [Fyodor] o The Nmap 5.00 tarball contains: -rw-r--r-- fyodor/fyodor 122943 2009-06-24 14:35 nmap-5.00/docs/scripting.xml -rw-r--r-- fyodor/fyodor 151 2009-06-24 14:35 nmap-5.00/docs/nmap-usage.xml -rw-r--r-- fyodor/fyodor 604 2009-06-24 14:35 nmap-5.00/docs/nmap-man-enclosure.xml -rw-r--r-- fyodor/fyodor 76918 2009-06-24 14:35 nmap-5.00/docs/nmap-install.xml -rw-r--r-- fyodor/fyodor 10179 2009-06-24 14:35 nmap-5.00/docs/legal-notices.xml If we're going to include the XML source files, we should include refguide too. But rather than add that, we should probably take these out. After all, people can easily grab them from svn or our new http svn gateway if desired. So no need to bloat the tarball with these files which aren't installed. [We're going to take the XML source files out of the tarball] [Fyodor] o Consider converting this file to emacs org-mode (http://orgmode.org/) format. [Fyodor] o That format is still plain text and can be read/edited by vi users, etc. [Considered, but I don't think I'll change right now] o Windows 7 RTM Nmap testing (With particular attention to 64-bit and our pcap installer). [Fyodor] o We should print host latency (when available) in the XML output, as suggested at http://seclists.org/nmap-dev/2009/q4/215. docs/nmap.dtd will have to be modified accordingly, and you might even consider adding support to docs/nmap.xsl. o Integrate latest OS fingerprint submissions and corrections. This was last done based on submissions up to May 8, 2009. o Potential OS X 10.6 problems. There are two issues reported by the same user which may be related: http://seclists.org/nmap-dev/2009/q3/0936.html, http://seclists.org/nmap-dev/2009/q3/0996.html. One is that Nmap hangs doing nothing and needs to be killed with Ctrl-C, and the other is that it dies after printing "Initiating UDP Scan". Another reported the same problem at http://seclists.org/nmap-dev/2009/q3/0990.html, where it dies after the first ARP request is sent. But Brandon has run Nmap on 10.6 without problems. It is a bit of a mystery. [David] [Resolution: Apple fixed the problems in 10.6.2; For users who have 10.6 and 10.6.1, the versions David builds on 10.5 will still work for them because they are 32-bit binaries rather than 64. Users who build Nmap on 10.6 or 10.6.1 should compile with -m32 or update to 10.6.2] o [NSE] Patrick's worker thread patch: http://seclists.org/nmap-dev/2009/q4/294 o Investigate get_rpc_results error (infinite loop) reported by Lionel Cons. See these threads: http://seclists.org/nmap-dev/2009/q4/24, http://seclists.org/nmap-dev/2009/q4/120 o Upgrade to latest version of NSIS on Nmap Win build system [Fyodor]. o Standardize on a proper file header for the Zenmap source code. [David] o For now, David is going to augment the templatereplacement system to insert the normal nmap.header.tmpl, but change the comment format to work with Python, and then replace the current Zenmap headers with that. o We may want to look into if/how we support IPv6 nameservers. Here is a bug report from someone having a problem with them: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=539244 [Ankur] o Once all the man page languages are in the Nmap tarball, we should update our install system to install them in the appropriate place. We'll want to integrate this with configure so users can decide which languages they want. See http://seclists.org/nmap-dev/2009/q4/249. o Resolve allow_ipid_match issue which can cause some malformed replies to be ignored when we might be able to still use them. See this thread: http://seclists.org/nmap-dev/2009/q2/665 [David] o Fix Zenmap 'make install' TypeError issue (http://seclists.org/nmap-dev/2009/q4/225). [David] o Fix a bug in which Nmap can wrongly associate responses to SYN and ACK host discovery probes. [David] For example: # nmap -sP -PS80 -PA80 australia.gov.au --packet-trace -d2 SENT (0.0760s) TCP 192.168.0.21:60182 > 152.91.126.70:80 S ttl=43 id=13466 iplen=44 seq=4046449223 win=4096 SENT (0.0770s) TCP 192.168.0.21:60182 > 152.91.126.70:80 A ttl=48 id=39976 iplen=40 seq=4046449223 win=1024 ack=921915001 RCVD (0.3020s) TCP 152.91.126.70:80 > 192.168.0.21:60182 SA ttl=53 id=0 iplen=44 seq=3924706636 win=5840 ack=4046449224 We got a TCP ping packet back from 152.91.126.70 port 80 (trynum = 0) ultrascan_host_probe_update called for machine 152.91.126.70 state UNKNOWN -> HOST_UP (trynum 0 time: 226875) Changing ping technique for 152.91.126.70 to tcp to port 80; flags: A In the example above, Nmap wrongly uses ACK as the preferred ping technique, when it should be SYN. [David] o we're thinking about ways to encode the information better. Right now we have pingseq and tryno, but we may want to just move to a single probe ID and then we can look up any other information in structures attached to that ID in memory when we get the response. o A related problem, which we hope the fix for this will also resolve, is that replies can currently match any probe whose tryno is less than or equal to the tryno encoded in the reply. o However, "fixing" this problem has been shown in the past to cause accuracy problems. See http://seclists.org/nmap-dev/2009/q1/387. We should figure out whether we can still reproduce that and, if so, what is going on before "fixing" this issue. o Add PJL (Printer Job Language) probes to nmap-service-probes. Brandon wrote some in http://seclists.org/nmap-dev/2009/q1/0560.html. Test them to see if they cause anything to be printed out (on paper) with printers that don't support PJL. If not, then remove the JetDirect ports from the default exclude list. The script pjl-ready-message.nse also uses PJL. We have concerns about the safety of this probe given http://seclists.org/nmap-dev/2009/q4/61, but it still is probably better to have the probe in there than not, as long as we continue blocking the ports by default with the Exclude directive. [We put in the probes, but are keeping the Exclude directives because the probes still seem a bit dangerous] o [NSE] in_chksum in packet.lua doesn't work with an odd number of bytes. Also make it more efficient. o Add --confdir option to Zenmap. See http://seclists.org/nmap-dev/2009/q1/92 [David] o Update our Winpcap from 4.0.2 to 4.1.1 (http://seclists.org/nmap-dev/2009/q4/128). This is a bit complex because we have our own installer. See https://nmap.org/svn/mswin32/winpcap/Upgrading-Instructions.txt. o Change Nmap to not show the "Host not scanned" lines in list scan o Change Nmap to show latency in "host is up" lines even w/o verbose mode. o Update our included Libpcap from 0.9.7 to 1.0.0 (http://www.tcpdump.org/) [David] o Improve Nmap output to show the forward DNS name when specified on command line as well as rDNS where appropriate. We're also going to reorganize output to enable some other improvements as well. See the proposal at http://seclists.org/nmap-dev/2009/q3/814, and that whole thread which starts at http://seclists.org/nmap-dev/2009/q3/805 [David]. o [Zenmap] Solve some unusual utf8 Zenmap crashes reported in the crash reporter. David has fixed some of them so far, but there are a few more remaining that may be related. [David] o Change Nsock to give an error if you try to FD_SET a fd larger than FD_SETSIZE. [Brandon] o Some research from David: We have help off on this change because of Windows portability problems. The Windows fd_set works differently than the Unix fd_set. In Unix, FD_SETSIZE (which is typically 1024) is both the maximum number of file descriptors that can be in the set and one greater than the greatest file descriptor number that can be set. In other words, we want to bail out whenever someone tries to FD_SET file descriptor 1060, for example. But on Windows it's different: FD_SETSIZE is only 64, but any file descriptor numbers, no matter how great, may be stored in the set. Windows socket descriptors are typically greater than 1023, but you can only have 64 of them in the set at once. So the fix on Unix would be --- nsock/src/nsock_core.c (revision 15214) +++ nsock/src/nsock_core.c (working copy) @@ -97,6 +97,7 @@ do { \ assert((count) >= 0); \ (count)++; \ + assert((sd) < FD_SETSIZE); \ FD_SET((sd), (fdset)); \ (max_sd) = MAX((max_sd), (sd)); \ return 1; \ @@ -107,6 +108,7 @@ assert((count) > 0); \ (count)--; \ if ((count) == 0) { \ + assert((sd) < FD_SETSIZE); \ FD_CLR((sd), (fdset)); \ assert((iod)->events_pending > 0); \ if ((iod)->events_pending == 1 && (max_sd) == (sd)) \ But that doesn't work on Windows (I just tried it) because even the smallest socket descriptor is bigger than FD_SETSIZE, 64. Really we're trying to accomplish two different things on the two platforms: On Unix we must not store a file descriptor greater than 1023, no matter how many or how few other descriptors have been set. On Windows we must not set more than 64 descriptors at a time, no matter what their descriptor number happens to be. o Add a way in NSE to set socket source addresses and port numbers. See this thread: http://seclists.org/nmap-dev/2009/q3/821. Some potential solutions are discussed later in the thread. o [Ncat] Fix --max-conns on Windows so that it only counts concurrent connections and not long-dead ones. See this thread (http://seclists.org/nmap-dev/2009/q3/1017.html) and particularly this message (http://seclists.org/nmap-dev/2009/q3/1032.html) for details. Venkat has a patch for David to review and potentially merge. o [Ncat] Fix 100% CPU usage with ncat -l --send-only. See this thread: http://seclists.org/nmap-dev/2009/q2/797 and continues further at http://seclists.org/nmap-dev/2009/q3/99. This message is key: http://seclists.org/nmap-dev/2009/q3/308 [David] o [Seclists] There is currently some extra vertical space after the first post of a thread in the thread index (example: http://seclists.org/nmap-dev/2009/q4/index.html). o [NSE] Decide which scripts belong to the "safe" category (we now have 20 which aren't either safe or intrusive), then remove the intrusive category since people can now specify "not safe". See http://seclists.org/nmap-dev/2009/q3/1091.html and that whole thread. [Fyodor] [ OK, see http://seclists.org/nmap-dev/2009/q4/0002.html] o [NSE] Fix http pipelining. Responses are being split on anything that looks like HTTP/1.X which doesn't come at the beginning of a line, and doesn't work when a line like that happens to legitimately come in a body. Joao has an nmap-exp branch which resolves this issue, though David found some bugs in that and sent some hard test cases. [Joao] o Fix traceroute performance/algorithms. It is terribly bad in some cases. For example, this traceroute scan took 36 minutes against a single host(!): http://seclists.org/nmap-dev/2009/q3/0425.html . We don't need to go up to hop 50 in such cases (maybe some heuristic like "at least go to hop 15, and stop after 5 unresolved in a row). And more importantly, there is no reason each hop should take 40s to timeout. It should probably use timeout variables like we use in port scanning. And it should parallelize as much as possible. Even if parallel resolution means we went a little further than we had to in incrementing the TTL, and we go to hop 15 when host is at 12 that's no big deal (of course we would only report up to hop 12 in the output). Once we do this, we should put back the ability to make --traceroute work even when we haven't found a probe which elicits a response from the target. (that feature was added in July, but we'll probably take it out until we can fix performance). [David] o Fix four Nmap bugs discovered by Ankur and analyzed a bit by David. [Ankur] o [NSE] Consider HTTP request caching. o [NSE] Finish (or write new) favicon fingerprinting script. See http://seclists.org/nmap-dev/2008/q4/0583.html . May need to do some more scanning and increase the DB size a bit. May or may not want to later combine this as part of a larger webapp fingerprinting script. o [Zenmap] When the inventory is changed, the current host/service selection is forgotten and the Ports / Hosts tab is switched to hosts mode. It should remember your current selection and not change the view. [David/SoC] o Device categorization improvements o Examine Nmap's device categorization in nmap-os-deb and nmap-service-probes. Decide if some small categories which have never really took off should be consolidated, or whether others should be split off. For example, maybe there are some groups in 'specialized' or other misc. categories which are now large enough to split off. Personally, I wouldn't give anything its own category unless there are at least half a dozen of them and no other category really fits them well. We should use a combined system for nmap-os-db and nmap-service-probes. o Add a classification sect1 to os-detection.xml (https://nmap.org/book/osdetect.html) to cover how Nmap handles OS classification. It should include a list with descriptions of each device type recognized by Nmap. Version-detection.xml should reference (link to) it in the approprate place. [Doug has done some initial work on this. For example, see nmap/docs/device-types.txt] [David] o Consider what new UDP payloads we might want to add. David has many ideas at: http://seclists.org/nmap-dev/2009/q3/0290.html o For traceroute we should give some indication that the RTT is in ms. Changing the column header to maybe "RTT MS" or "RTT (MS)" would probably do the trick or we could append "ms" to each value. [David] o OS fingerprint should probably specify somewhow when DS=1 if it's because target->directlyConnected is true, or because it sent the distance probe and calculated a distance of 1. The second situation should never happen, but often David strongly suspects that it is the case. o --traceroute should probably set currenths->distance because right now, I do an -O scan against scanme.nmap.org, and it does not figure out the distance. So the fingerprint shows no distance element and Nmap doesn't print "Network Distance" in the results line. That may be OK (Nmap probably isn't receiving the probe response needed for this, and maybe doesn't want to print the TG), but even when I do --traceroute I get no distance printed. Yet Nmap clearly knows the distance since the traceroute shows all the hops up to and including the target (scanme.nmap.org). o Figure out best favicon to use for Nmap and related web sites [David] o [Ncat] David says: "After you get EOF on stdin with --send-only, the program hangs on until the idle timeout expires instead of terminating immediately. I had a fix for it but it involved deleting events in the Nsock queue and it caused an assertion failure in Nmap so I backed it out. I have a less intrusive solution." [David] o We should update our config.{sub,guess} files. This Debian bug #542079 requests that we do so: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=542079. We last updated on 3/15/08 and in that case we used versions from http://cvs.savannah.gnu.org/viewvc/config/?root=config. That may or may not be the best place to get them now (e.g. perhaps there has been a recent official release). [David] o Look a bit more at default version detection timing. Particularly deciding the number of probes to run in parallel. [ We increased that a bit on 8/18/09] o [Ncat] Right now our -i (idle timeout) causes Ncat to quit if EITHER reading or writing is idle for the given amount of time. But it is really only idle if BOTH reading AND writing are idle for the period. We should make the code work that way. o Add scripting.xml documentation on strict.lua and the avoidance of global vars in libraries. See http://seclists.org/nmap-dev/2009/q3/0169.html. Probably a new section just above "Adding C Modules to "Nselib", such as "Writing Your Own Library" or somesuch. [Patrick] o Update nsedoc to refer to 'libraries' rather than 'modules'. This affects the front page (which calls them 'Libraries' on left sidebar and 'Modules' on the list of right, and affects the url (we should change /modules/ to /lib/ and then have Fyodor add a redirect for people still using old URLs) and the title of the module pages like https://nmap.org/nsedoc/modules/base64.html. [Patrick] o [Ncat] Prefix Ncat stderr messages with "Ncat: " to make it clear that they are coming from Ncat and not the remote server (or typed in by user). [David/SoC] o [NSE] Optimize NSE Performance--e.g. measure the current performance and see what can be improved in terms of scheduling scan threads, determining how many to run concurrently, looking at CPU load items, etc. [David/Patrick] o Increase version scan concurrency based on Patrick's performance testing. We decided to go to 20 for timing_level 3, 30 for 4, and 50 for 5. o [NSE] Consider POST/HEAD support. See http://seclists.org/nmap-dev/2009/q1/0889.html. o Implemented: http://seclists.org/nmap-dev/2009/q3/0074.html o Joao going to check in very soon soon. o [NSE] Consider Rob Nicholls http-enum script for incorporation: http://seclists.org/nmap-dev/2009/q1/0889.html [Joao tested w/his HEAD support, is going to check this in] o Consider the open proxy scripts more carefully - How should we test whether the proxy attempt was successful? Right now we look for a google-specific Server header after trying to reach http://www.google.com through the proxy. Maybe we should let users specify their own pattern if they specify their own URL. [ Joao is going to check it in today (7/28)] o I should add code to Nmap to bail if sizeof(char) isn't 1. Otherwise there could be security risks if it is not one on any platforms. [ Actually, we think C standard requires this and we've not heard of any system where sizeof(char) isn't 1. So removing this item.] o [Zenmap] More complete implementation of ZenmapCommandLine/profile editor improvement ideas. See http://www.bamsoftware.com/wiki/Nmap/ZenmapCommandLine. [David] o [Ncat] Think about whether we should offer "-q secs" (quit after EOF + delay of secs) and/or -k (set SO_KEEPALIVE on socket) (or maybe that should be set by default). Anyway, these were suggested here: http://lwn.net/Articles/341706/ [We're going to fix -i (added separate item), and not worry about SO_KEEPALIVE unless we see more demand for it. It doesn't seem that nc110 or OpenBSD nc or so-called GNU Netcat support SO_KEEPALIVE either] o [Ncat] In verbose mode, I'd like to see clock time (duration) and maybe in/out traffic stats when a client connection ends. Maybe it could use a format similar to what Nmap provides. [David/Venkat] o Seriously consider making --traceroute work even when we haven't found a probe which elicits a response from the target. We'd just have to pick a probe in that case (probably echo request, as we found that to be the most effective in prev. empirical testing). This is similar to UNIX traceroute and Windows tracert.exe which just pick a probe (high UDP port on UNIX, ICMP echo request on Win). Even if the host is down or something, we usually get some useful hop information. o [NSE] Allow spaces in script arguments without the user having to manually quote them (beyond normal shell escape quoting). See: http://seclists.org/nmap-dev/2009/q3/0090.html [Patrick] o [Ncat] Support SCTP now that Nmap does. - See client support patch by Daniel Roethlisberger: http://seclists.org/nmap-dev/2009/q2/0609.html - Server support? - Daniel has a patch, David looking to apply once an nsock thing is fixed. o Look at etc/payloads.conf in unicornscan-0.4.7 and see if they have any which we don't have, but should, for our version detection. They have a decent collection there. KX sent some other programs we should look at too. [David] o Ncat should give it's ethernet cat ASCII logo after configure--similar to the way that Nmap, Ncrack, and Nping do. [David/SoC] o [Zenmap] The Search dialogue is helpful for finding a certain scan you've performed recently, but we should probably also offer a similar function for searching for certain applications/hosts within a scan (e.g. find all the hosts running Apache). This new functionality might be a find option or some other mechanism rather than being part of the Search dialogue proper. o Ncat SSLv2 issues. See http://seclists.org/nmap-dev/2009/q1/0319.html. A big part of it is done, which was enhanced version detection probes to detect more SSL servers, The defect that remains is that Nsock can't connect to a small fraction of servers (including some of the ones detected by the new version probe). They are the servers that do only SSLv3 or TLSv1 and don't respond to a SSLv2-compatible ClientHello. Even though most servers don't support SSLv2, they usually respond to the ClientHello and just don't offer any SSLv2 features. [David/Venkat working on this] o Deadlock identification and correction: o Plan of action: implement freeing of script mutexes when scripts exit without freeing them (done and in /nmap now). And then if it continues to be a problem we'll consider this other stuff: o Add detection for deadlocks and print which threads are involved. o use above results to make a strategy for automatic deadlock resolution. o Original entry: Figure out what to do about NSE mutexes: http://seclists.org/nmap-dev/2008/q3/0276.html . In particular, they are not currently cleaned up if a thread dies or otherwise exits without unlocking them and can cause endless deadlocks which are annoying to users and can be difficult to debug :(. Patrick has some ideas for this in his SoC09 proposal: "Adding a cleanup system for NSE that is called periodically similar to nsock_loop. There would be a registration system allowing C libraries to register a Lua function that will run periodically to check for irresolvable deadlock or simply dead resources. For example, the nmap library would register a mutex cleanup handler which would inspect all mutexes looking for a dead thread or circular dependencies. The nsock library could register a handler that checks for unused sockets. The nsock may save a strong reference to the thread that owns the socket and inspect it to determine if the thread is dead." David later says: "After some discussion we decided to start more modestly, first by ensuring that a scripts mutexes are released when it dies for whatever reason. I have a hunch that this is the cause of most deadlocks. It was certainly the cause of two whois.nse deadlocks I found. Then, the next step if deadlocks continue to be a problem, is to do automatic detection and just print out a list of what scripts are involved. It could be that several smb scripts are deadlocked, or as in the case I observed where whois.nse was locked with itself." o Joao is auditing his Lua code to make sure all his variables are local where appropriate. [Joao - done, should be commited very soon] o [NSE] We need to deal with libraries which improperly use global variables, as that is very common (Patrick made a list: http://batbytes.com/bad.txt). Solutions could involve augmenting our runtime system (the "strict.lua" approach) to detect/prevent the problem, a script we run occasionally to identify issues that we then manually resolve, or, at the very minimum, documenting somewhere in scripting.xml the dangers inherent in global variables and warn people to generally declare them local instead. We have a long history of bugs caused by non-local variables defined in NSE libraies and often causing deadlocks. o The Nmap refguide (https://nmap.org/book/man-performance.html) says "The --max-parallelism option is sometimes set to one to prevent Nmap from sending more than one probe at a time to hosts. This can be useful in combination with --scan-delay (discussed later), although the latter usually serves the purpose well enough by itself." But when you actually try it: # ./nmap --max-parallelism 1 --scan-delay 10 scanme.nmap.org You can't use --max-parallelism with --scan-delay. QUITTING! We need to either make that work or adjust the documentation. [David/SoC] o David changed this to a warning. Note that with --scan-dealy, --max-parallelism is essentially 1 anyway. o [NSE] Consider integrating HP Laserjet print PJL status-setting script. See this thread for an example of such a script: http://seclists.org/nmap-dev/2009/q3/0083.html (note that it is updated during the thread). Also, see this thread: http://seclists.org/nmap-dev/2009/q3/0092.html o Ndiff man page should be expanded to include sample execution/output and more fully describe its functionality. [David] o David is going to reexamine the old coverity-reported issues (the ones we previously marked as "ignore" because they weren't real bugs) just to be sure that is (and is still) the case. o Make -sP work with -PN to disable both port and ping scanning. We need to make sure the various options still work (-O, --script, --traceroute, etc.) with this, as many currently don't as they don't expect this behavior, which used to be unsupported and cause Nmap to quit with an error messaqge. It may be OK to refuse -O since that will rarely give useful results. OTOH, -O may work on some systems with unique closed port signatures where Nmap guesses a closed port. Users should then be able to do an NSE-only scan with "-sP -PN --script [scripts]" We should document this -sP -PN usage in refguide. [David] o Add -sn and -Pn options which are aliases for -sP and -PN. Once they've been around long enough to be in most people's copy of Nmap, we plan to document those as the preferred version. Those match -n, and the main problem with -sP is that we now use it more for "disable portscan" than ping only. For example, you still might want to use NSE. [David] o [NSE] Make sure all our HTTP scripts transparently support SSL servers too. [Joao has a solution and is testing the http scripts to make sure they don't break.] o Resolve "memcpy overlap in getinterfaces(int*) (tcpip.cc:2987)". See this thread: http://seclists.org/nmap-dev/2009/q2/0713.html [David/Brandon] o [Ncat] Print a message to stderr upon connection failure even if -v isn't specified so the user knows what went wrong. [David/SoC] o [Ncat] Maybe --chat should imply -l. And Maybe --broker should too? - OTOH, we might want to extend --chat for connect mode in the future. [We're going to hold off on chat now, David/SoC is doing --broker] o Consider making it easier to tell whether scripts were specified by name on the command-line (rather than default or by class) so they have the option of providing extra verbosity in that case. For example, see http://seclists.org/nmap-dev/2009/q2/0563.html. We could either provide a special function for scripts to determine that, or we could magically adjust nmap.verbosity() when called by those scripts. [David] o [NSE] Figure out a way to support people who want to do script scan, but not port scan or ping scan. One option would be to allow --script to list scan (-sL), but perhaps a better option is to provide a way to disable port scanning in the same way as we offer -PN to disable ping scanning. As an example of this need, David had to write special code to avoid ping/port scanning when doing a whois.nse survey for http://www.bamsoftware.com/wiki/Nmap/EffectivenessOfPingProbes. The key for this task is to figure out how to do it from a user interface perspective and then implement and document it. We've already been going in the direction of allowing script scanning in more types of scans--a while back we started allowing it with -sP ping scans due to high demand. [David/SoC] [ We decided how we're going to do it (-sP -PN to start out with; leading to eventual -sn -Pn) and added new TODO entries for actually doing the code/docs. ] o Ndiff should be able to show NSE script result changes. [David] o Get set up for Coverity scan of latest version to see if it catches any important issues before stable release. [Fyodor,David] [Found 7 new results, 3 are real bugs, and 2 have been fixed so far] o [nsock] Fix Makefile to handle dependencies correctly (if that turns out to be the problem). See http://seclists.org/nmap-dev/2009/q1/0629.html. o Or it may be related to SVN timestampling. See http://seclists.org/nmap-dev/2009/q1/0632.html. Diagnosed by David: http://seclists.org/nmap-dev/2009/q2/0728.html o For at least our UDP ping probes, Nmap should probably notice if it is a very well known service port such as 53, 161, or 137 and send an appropriate probe packet (server status for DNS, public community string query for SNMP, etc) rather than empty data in that case. This is similar to the way our IP protocol probes automatically include common headers such as TCP and UDP if that common protocol is given. Good probes for these services are already available in nmap-service-probes, though we might want to make a custom file for this. We should probably do this for port scanning as well. [David] o [NSE] Make NSE work better for SSL tunneled services in general by supporting them easily in the libraries. For example, I don't think irc-info.nse currently works against all the servers which tunnel over SSL. Maybe augment comm library, etc. [Joao - done, except for http, which is already a separate TODO item] o Update scripts which use table args to use pseudo-table format "name.arg" rather than requiring the user to create a Lua table themselves. On the lua side, it's not really being stored in a table, but just an arg named "name.arg". [Joao] - Look at all our existing scripts which use tables (dns-zone-transfer, whois, the proxy scripts, etc.) and change as appropriate. Remember to change the usage throughout the script and also change the nsedoc script arguments and example usage. For the existing scripts, try to retain the table version check for now to avoid breaing backward compatability if possible. Just add the newer style check as well. - Is taking arguments in a table specific to a script a good idea? The example in the socks-open-proxy nsedoc of "--script-args openproxy={host=}" is a bit of a mess and I'm not sure the best way to document that in the script argument list. Note that this is the standard way we've handled it for some other scripts, so it's not an open-proxy-script-specific problem. o [NSE] Track active sockets in the nsock library binding and don't rely on garbage collection for reallocation. Can probably wait until post-stable release for integration. [Patrick] - Patrick has a patch and is waiting on dev branch to check it in. o [NSE] Resolve ssh2.lua buffering problems (http://seclists.org/nmap-dev/2009/q2/0673.html) [Joao] o Decide what to do about ncat source code headers -- maybe just use the Nmap ones. [David added the Nmap headers] o Once we go into deep stability freeze mode, create an nmap-exp development branches for changes we plan to integrate after the stable release. [Fyodor] o Update CHANGELOG for latest changes [Fyodor] o Release 4.85BETA10 o [NSE] Open proxy detection scripts o We have http-open-proxy.nse, but we should probably either extrand that to handle other types of proxies (such as SOCKS and HTTP CONNECT) or create more scripts to handle those other proxy types. [Joao, David] o Joao has written scripts, just need to finish up, evaluate, integrate. o Determine whether zenmap.spec.in can currently require "python-sqlite" rather than "python-sqlite2", or if it at least can be easily made to do so. The former seems more compatible since RHEL/CentOS 5.3 has a "python-sqlite" package, but not "python-sqlite2". Meanwhile, Fedora 10 provides the "python-sqlite" capability as long as you have the Python 2.5 package installed (python-2.5.2-1.fc10). Fedora 10 does also make a python-sqlite2 package available. o [Ncat] Solve EOF issues which crop up when piping to an external command. See http://seclists.org/nmap-dev/2009/q2/0528.html. It sounds like we will go with Daniel's patch [Daniel, David] o Look into building RPMs with SSL support. Statically linking to OpenSSL on Linux for the RPMs didn't work for me last time I tried. [Fyodor] o Static linking of Nmap to OpenSSL does not seem to work on Fedora 10 or CentOS 5.3. The problem appears to relate to the OpenSSL krb5 support. o Could build my own OpenSSL libraries on the build system (w/o Kerberos support) and link to those. o At some point, we might want to consider including OpenSSL with Nmap tarball. The problem is that it is rather big. Would increase Nmap .tar.bz2 size from about 9 megs to about 12. OTOH, OpenSSL is only going to get more and more important. Maybe we can include a stripped down version? o If we don't integrate OpenSSL (or until we do), we might consider a more prominent configure warning for when SSL is not detected. We could suggest that users run "yum install libopenssl-devel" or "apt-get install libssl-dev" commands or whatever is appropriate and then reconfigure. Or we could point them to a page or nmap-dev posting URL with instructions. o Figure out why I [Fyodor] get a bunch of "Operation not permitted" errors when I launch a scan on SYN such as: - I'm going to ignore this for now unless it causes me trouble again, as this is an old machine that will be replaced soon anyway. And we haven't been hearing of the problems from others lately. /home/fyodor/nmap-exp/fyodor-perf/nmap -nogcc -T4 -n -v -p- --portpingfreq 250 -oA /home/fyodor/nmap-misc/logs/WorldScan/portpingfreq/logs/portpingfreq-250-1%T-%D 67.15.236.34 67.15.236.36 81.174.236.66 81.174.236.119 170.140.20.160 170.140.20.174 202.138.180.9 202.138.180.17 202.138.180.132 209.20.64.112 The errors look like: sendto in send_ip_packet: sendto(7, packet, 44, 0, 170.140.20.174, 16) => Operation not permitted Offending packet: TCP 64.13.134.4:59820 > 170.140.20.174:59120 S ttl=39 id=19927 iplen=44 seq=2425535549 win=4096 sendto in send_ip_packet: sendto(7, packet, 44, 0, 67.15.236.36, 16) => Operation not permitted Offending packet: TCP 64.13.134.4:59820 > 67.15.236.36:15030 S ttl=57 id=50640 iplen=44 seq=2425535549 win=2048 Discovered open port 49394/tcp on 170.140.20.174 sendto in send_ip_packet: sendto(7, packet, 44, 0, 170.140.20.174, 16) => Operation not permitted Offending packet: TCP 64.13.134.4:59819 > 170.140.20.174:8256 S ttl=48 id=38510 iplen=44 seq=2425601084 win=1024 May be related to connection tracking and high scan rates. See http://seclists.org/nmap-dev/2008/q4/0652.html http://www.shorewall.net/FAQ.htm#faq26 Others have reported similar issues even without connection tracking. See http://seclists.org/nmap-dev/2006/q3/0277.html http://seclists.org/nmap-dev/2007/q2/0292.html o -PO1 and "-sO -p1" seem to send ICMP ping packets with an ICMP ID field of 0, which we found that a small percentage of hosts drop (61.13% responded with 0, 62% with a random value). So we might as well randomize them in these cases. [Josh Marlow] o Some of the -PS443 scans (and maybe other ones) we've been running have been missing the Nmap line telling how many packets were sent/received, even though we had verbose mode. [David/Josh] o Deal with Ncat newline problem. See this thread: http://seclists.org/nmap-dev/2009/q2/0325.html [David,Jah] o Integrate SCTP scanning support. See Daniel Roethlisberger's branch in nmap-exp/daniel/nmap-sctp. As of 4/30/09, he is nearing completion. See http://seclists.org/nmap-dev/2009/q2/0270.html. o [NSE] Release mutexes upon script death to prevent certain deadlocks [Patrick, David] o Consider whether to let Zenmap Topology graph export the images to svg/png/etc. Also think about printing. Note that João Medeiros has written a Umit patch to do this: [Joao, David] http://trac.umitproject.org/ticket/316. - Now he has Nmap patch: http://seclists.org/nmap-dev/2009/q2/0409.html - Consider integrating. - Integrated! o Ensure that when I build a distribution package on UNIX (e.g. make distro), it builds what is in the Nmap directory I am calling it from rather than a particular SVN version. I'm going to start building packages from a special "clean" directory which is different than the one I do development work in. Also, I want to be sure that any changes in that dir are included in the release, even if they aren't check in yet. [Fyodor] o Nmap UNIX distro build script should regenerate script.db. [Fyodor] o Now it is in make prerelease o Nmap build system should be split into [Fyodor] o prerelease -> generates version files, man pages, script.db etc. That has to be done on one system, and then results checked in before doing a make release. It does this stuff based on the directory it is run in rather than some set dirname or a pure SVN version o release-tarballs -> does any system-dependent building and creates the source tarballs. It does this stuff based on the directory it is run in rather than some set dirname or a pure SVN version o release-rpms -> Same as above, but also uses the created tarballs to build the Linux RPM binaries for the current platform based on the tarballs. o Build x86 and x86-64 VM instances for RPM building. [Fyodor] * I think I'll use CentOS 5.3 o [NSE] Script scanning does not seem to work on Fyodor's Linux machines after being installed from latest SVN (or 4.85BETA9) and run as a non-root user (it works fine as root). The command "nmap -sC localhost" leads to NSE failure messages which differ based on the exact version run. [Was a relatively simple permissions problem in our Makefile.in -- I fixed it] o [NSE] Release socket locks on connection failure or timeout. [Patrick] o Update Nmap entry on Linux Online - http://www.linux.org/apps/AppId_1979.html - Screw it, the site does not seem to be maintained at all. They aren't taking updates as of 6/2/09, and even Firefox shows latest update as 0.9.1. o [Ncat] In verbose mode, print when an SSL connection is established successfully and give the leaf certificate hash to make it easier to verify when connecting to a machine where you can't or don't want to use --ssl-verify (e.g. connecting to an ncat ssl server where it created its own key). While we're at it, we might want to print some other information from the leaf node, such as organizationName and maybe localityName, countryName or something. We don't want to be too verbose, but 1 line would be great and 2-3 might be acceptable. [David] o Fix NSEdoc to better escape single-quotes in fields. If we can't do that for some reason, we need to document it better. For example, when we initially tried generating nsedoc for http-webdav-unicode-bypass.nse, NSEdoc was listing it as a module named "s auxiliary module", apparently because this line exited in the description field: This module is based on Metasplit's auxiliary module, modules/auxiliary/scanner/http/wmap_dir_webdav_unicode_bypass.rb. (For full example, see scripts/http-webdav-unicode-bypass.nse r13345) [David/SoC] o --script-args should allow a wider range of characters, and should give a more useful error message if it receives chars it really can't handle for some reason. For an example, try "--script-args=smbuser=admin,smbpass=pass^word". For more details, see Ron's report at http://seclists.org/nmap-dev/2009/q2/0378.html. o [Ncat] Have --ssl-cert and --ssl-key send a certificate in connect mode so that client certificate auth can be done. [David/Venkat] o Once we're done with host discovery empirical research, add it to host-discovery.xml. Would be great to show the best combinations to use for a given number of probes, the efficiency of the common probes by themselves, etc. o Consider making the ping scan default be more comprehensive. Note that I got 23% more Internet boxes found out of a 50K sample (see host enumeration chapter of my book for details). Maybe I should experiment a bit more to ensure they are real boxes and not network artifacts and figure out exactly which tests are helping the most. If I do this change, I'll have to update the host enumeration chapter. For UDP probing purposes, we should test whether including extra data in the packet (e.g. --data-length) helps in general, and for services such as 53 and 137, we should probably send proper protocol headers (e.g. a DNS server status message) so that we receive responses from listening services. o We should probably check for a system Lua in a "lua5.1" directory rather than just "lua", as Debian and also my Fedora 10 systems seem to have that. See http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=527997. [Note, Fyodor asked the bug reporter Jan Nordholz on 5/14/09 if he could write a patch. Jan sent in a patch, it worked, Fyodor checked it in.] o [NSE] Get rid of ceil so that floating point NSE runlevels work again (some scripts, including (smb-brute) rely on this. They got broken with the NSE core lua rewrite. [David]. o NSE script logical operator stuff is now documented in scripting.xml--add to refguide.xml as well. [David/Patrick] o [NSE] Correct nsock_connect to unlock the socket slot if the connection fails. When a socket is closed, it is unlocked so the arbitrator can potentially open up a socket for another thread. But Patrick discovered that a socket is not automatically unlocked when a connection fails or times out, only when it is closed explicitly. So that could hold up socket allocation for other threads until garbage collection. May be a cause of slowness or possibly deadlocks. [Patrick] o [NSE] Solve segfault issue which occurs when Nsock events call back on a thread that has already ended (e.g. timeout, crash, early exit, whatever) and been garbage collected. May want to just nsi_delete all nsock sockets immediately upon thread ending. For an example of this type of segfault, see http://seclists.org/nmap-dev/2009/q2/0289.html. David says " I think in the interests of getting this in a stable release, we should use that strategy of closing all a thread's sockets. That ought to fix all the problems above. Not to rule out a more thoughtful redesign in the future." [David,Patrick] o We added the SEQ.CI value in Feb 2009 with 0 matchpoints. At some point (once we have some real-life values) we need to evaluate whether we want to give it points. A good time to do that would be when we next do fingerprint integration, so we will actually have examples of .CI in the nmap-os-db. [David] o [NSE] Make it a warning rather than error if a script in script.db can't be found. [Patrick] o Add version detection signature for Ncat chat once we finalize the announce format. [David] o Change Nmap signature files to use the .sig extension rather than .gpg.txt, as that seems to be what gpg recommends. In fact, gpg will automatically verify the right file if it exists after dropping the .sig (or .asc) extension. I may need to configure .htaccess to serve .sig files properly. Update nmap-install.xml accordingly. Suggested by tic at eternalrealm.net by email on 7/13/08. [Fyodor] * Rename existing files, add symlink from the old .gpg.txt to .asc versions * Add appropriate .htaccess content type if needed for downloads - not needed since I decided on .asc extension rather than .sig * Update the generation scripts * Update the book documentation - https://nmap.org/book/install.html#inst-integrity o Ask Coverity if they'll scan latest version of Nmap. [Fyodor asked David Maxwell on 5/14/09 ] o Make 4.85BETA9 release [Fyodor] o [Zenmap] Make a way to start a scan from the profile editor without creating a profile, then remove the command wizard. This is partial implementation of http://www.bamsoftware.com/wiki/Nmap/ZenmapCommandLine. [David] o [Ncat] Make proxy server mode work on Windows (this is the last remaining fork() dependency in Ncat). o Do an OS detection integration run -- last was based on 1/8/09. [David] o [Ncat] Maybe we should create an SSL cert with no passphrase during Ncat compilation or install process so that if someone specifies Ncat -l and --ssl with no --ssl-cert and --ssl-key, we already have one for them, and it is a slightly better one (since the private key isn't known) than if we distributed a key. Obviously it is still subject to MITM attacks since there is no domain validation going on. But people who need that will have to buy a key from a certificate authority in any case. We could create the key by using the "openssl" command line tool as shown in https://nmap.org/ncat/guide/ncat-advanced.html#ncat-ssl, or maybe better to have a way for ncat to do it using openssl calls. [David] o [Zenmap] Should probably give some sort of widget indication that a scan is running. Now that we can start multiple scans at once, the "scan" button goes back to being unpressed while the scan is running. As some scans take minutes or more to show output, it is not always clear whether they are still properly running. We should probably have some sort of widget, such as the throbber used in web browsers, to show that Nmap is still running. It could be fore a specific scan (kind of like how you have a separate throbber for each tab on a web browser), or a global one which means at least one scan is running. Or maybe a different sort of indication is in order (like a timer). [David] o Further investigate Nmap Proxy patch by Zoltan Panczel and Ferenc Spala. See http://nmap-dev.fw.hu/ and http://seclists.org/nmap-dev/2009/q1/0255.html . [Discussed it and then added new proxy feature item] o Wherever practical, fix compiler warnings when compiling Nmap with VC++ 2008 Express SP1 (there aren't many). [David] o [NSE] Consider adding boolean expressions to --script arguments. For example, see Patrick's implementation at http://seclists.org/nmap-dev/2008/q3/0300.html . o Generate a list of trusted SSL certificates to ship with Ncat (by extracting f rom Mozilla or similar), and install them with Ncat. Decide how these certificat es should be preferred to any system-provided certs, if any. [David] o [NSE] Add desired SoC09 infrastructure ideas to this TODO to the extent they don't already exist. o [Ncat] Consider supporting server certificate verification when used in client SSL mode. o For now we document in user's guide that it is not secure. o Maybe we can do an ssh-style approach where we just print the fingerprint and expect the ncat client user to ensure it is the right one? o If we're going to verify cert's etc., we need to also make sure we are actually using secure ciphers. We may need to update nsock to support cipher selection, because we want fast ones for version detection, but usually want secure ones for NSE and/or ncat. o Do we want to check all this by default, or offer an option for it? Doing it by default is more secure, though it can be annoying when a certificate has expired, is self-signed, you connect to domain.com when the certificate is for www.domain.com, etc. If it is done by deault, we might just print an error message. Whreas if we have a special option, it may be OK to exit and refuse the connection. o What certs should we allow? Same as the browsers do? Maybe get rid of Comodo? Maybe we should fail to recognize any certs with MD5 in the trust chain? o What about people who are running their own SSL service and just want to specify the cert file they use, because they generated it themself and not from a trusted CA. o Need to check expiration, domain, etc. if we're checking certs at all. o We can probably get away with not doing revocation checking, as long as we document that we don't. o consider changing status field from "up" and "down" to "online" and "offline". Actually, maybe we don't want this after all. online/offline look pretty similar, and they're longer too. I'm taking this out of the TODO. o [Ncat] When acting as an HTTP proxy, we should support GET mode as well as CONNECT so that it works as a non-SSL proxy in browsers such as firefox. [David] o Finalize GSoC applicant research, communication, and selection [David, Fyodor] o Go through all the SoC applicants and decide who we want to accept and start communicating with them. [David,Fyodor] o Decide which applicants we want, and who would be best for mentoring them. o Document that U1.RID gives "G" as long as all the data bytes in the echoed response data are "C" as expected. This G code is still given even when the response is truncated, including if there are 0 bytes echoed. [David] o [Ndiff] Rethink the output format. David says: In particular, I would like to always have the old state on the left and the new state on the right: "was filtered, is open," not "is open, was filtered." I also like the context diff output of MadHat's nmap-diff. [David] o Canonicalize the "host up" messages for port scan and ping scan so that instead of things like "Host scanme.nmap.org (64.13.134.52) appears to be up ... good." we standardize in both cases on something like: "Host scanme.nmap.org (64.13.134.52) is up (.75s latency)". Note the addition of the latency value, which is our srtt value for the host. This will only show in ping scan and verbose port scan because the line doesn't appear without verbose mode. [David] o Ping scans always seem to say "0 [hosts] undergoing Ping Scan" when you request stats, rather than the proper number. For an example, try a command such as "nmap -iR 10000 -sP -n" and then press enter during the scan. Here are some examples of the bad output: Stats: 25:34:33 elapsed; 991232 hosts completed (72530 up), 0 undergoing Ping Scan Ping Scan Timing: About 53.69% done; ETC: 22:49 (0:00:09 remaining) Stats: 0:01:10 elapsed; 0 hosts completed (0 up), 0 undergoing Ping Scan Ping Scan Timing: About 24.03% done; ETC: 22:42 (0:03:41 remaining) Stats: 0:03:28 elapsed; 4096 hosts completed (284 up), 0 undergoing Ping Scan Ping Scan Timing: About 3.06% done; ETC: 22:44 (0:03:07 remaining) [David] o Remove obsolete tests from nmap-os-db itself. [David] o Prepare for Summer of Code * Brainstorm for ideas * Create new ideas page * Apply to participate in program again * Advertise for applicants * Evaluate applicants o NSEDoc script/module documentation pages should probably provide a link to the script/module source code (except for C modules). The link format should probably be of the form https://nmap.org/data/scripts/[script].nse and /data/nselib/[module].lua. NSEdoc can assume they already exist there, as we'll probably put them there using the same system we use to copy other stuff to the data dir. o [Ncat] Let people set up authenticated proxies using --listen and --proxy-auth together (right now we don't support that). [David] o When you specify multiple comma-separated arguments to --script, those arguments seem to get lost when the Nmap command is printed in Nmap's output files. For example, I run the command: nmap -oN - --script=discovery,intrusive scanme.nmap.org The output includes: # Nmap 4.85BETA4 scan initiated Thu Mar 26 15:40:05 2009 as: ./nmap -oN - --script=discovery scanme.nmap.org Note the missing ",intrusive" in the script argument. [David] o Merge patrick/nse-lua-merge for easier-to-maintain and simpler codebase once David and Patrick are happy with it. [David] o SVN check out /nmap as an external in a directory named svn or src or nmapsvn or something under nmap.org web tree. Then redirect the individual nmap.org/data/ files, where needed, to the nmapsvn instead. and update nmap-dev Makefile not to copy them to the /data/ dir anymore. Then update the nsedoc system to generate proper links to the new script/nselib locations. [Fyodor] o Improvements to presentation of version detection information. [Brandon] o Allow longer strings. Right now it can be 128 chars for the fullversion info, I think. But that isn't enough for this useful information-packed string: "Apache httpd 2.0.52 ((Red Hat) mod_perl/1.99_16 Perl/v5.8.5 DAV/2 mod_jk/1.2.19 PHP/4.3.9 mod_python/3.1.3 Python/2.3.4 mod_ssl/2.0.52 OpenSSL/0.9.7a)". After discussion w/Brandon, we're going to allow 160 chars total. o Instead of omitting all information when version info string too long, we're going to truncate and allow 157 characters, plus ellipses (...) o Brandon says: "my final gripe is that the full version string is constructed as (). but, even if product or version are blank, the spaces are still there" o I need an output-autoflush option of some sort. This could be useful to ensure I get all the --packet_trace and debug data before Nmap crashes. Actually, I'm not sure that is so critical. o Killing it for now, not sure that it even is needed. o Fix the directory function(s) in nse_fs.cc to be usable by scripts and improve flexibility. [this entry added by Patrick] o [Ncat] The sys_wrap.c/.h code contains a whole bunch of capitalized versions of system calls (Fork(), Socket(), Sscanf(), etc.) which are mostly the same as the standard version except that they cause ncat to quit if they are triggered. They also may be used partially for portability. The main issues are: 1) Because the function quits in the case of errors, it doesn't always have the context to print a useful error message (and even when it does, it often doesn't -- for example Fopen could print the filename, but doesn't.) Also, sometimes these functions are called when quitting really isn't the desired outcome of an error. 2) Some could be replaced by code in nbase, for example, Malloc basically does the same thing as our safe_malloc already used throughout Nmap. So we should probably consider simplifying/removing this code to the extent possible. But we need to remember to add error detection to the callers where necessary rather than blindly switching from (e.g.) Connect() to connect(). [Kris or David] o With --version-trace (may be a problem with other uses of nsock tracing too), I often get dozens of "wait_for_events" reports in a row in a very short period, flooding the logs. For example, with the command "nmap -sV --version-trace www.google.com", I get: NSOCK (22.3570s) Callback: WRITE SUCCESS for EID 283 [74.125.19.147:443] NSOCK (22.3570s) msevent_delete (IOD #4) (EID #283) NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events NSOCK (22.3570s) wait_for_events [Goes on for pages] o NSE memory issues (and gh_list assert failure) [David] o See this thread: http://seclists.org/nmap-dev/2009/q1/0532.html o We're taking this out for now since the new nse-lua-merge tenatively looks like it fixes this. o [Ncat] Why does Ncat require enclosure in a while loop to answer repeated UDP queries, but not TCP? For example, see the "Emulating Diagnostic Services" section of the Ncat user's guide. o Note: http://seclists.org/nmap-dev/2009/q1/0133.html o Determine what we should do about the IE.DLI OS detection test [David] o All of the 1656 results for this test in nmap-os-db are DLI=S. o Is the test not working right (producing the proper results against targets), or is it just a generally useless test for which virtually all targets respond the same way? o Are there other "useless" tests in nmap-os-db? It is worth checking, IMHO. o We're going to get rid of IE.DLI, IE.SI, U1.RUL, and maybe TOS and TOSI tests. o When you do ncat -h, Ncat should probably show the Nmap version number rather than (currently) 0.2. Also ncat in -v mode should show that same header. [David] o Ncat verbose mode (-v) should probably only give important messages, such as perhaps a message once you connect successfully to a port, or a message if the connection attempt times out. An Ncat version banner (with URL) like Nmap has might be warranted (in verbose mode). Currently, Ncat floods you with (mostly) useless debugging information like this with a single -v (this output, on the other hand, might be useful for a debugging option): [David] # ncat -C -v scanme.nmap.org 80 NSOCK (0.0000s) TCP connection requested to 64.13.134.52:80 (IOD #1) EID 8 NSOCK (0.0200s) Callback: CONNECT SUCCESS for EID 8 [64.13.134.52:80] NSOCK (0.0200s) Read request from IOD #1 [64.13.134.52:80] (timeout: -1ms) EID 18 NSOCK (0.0200s) Read request for 0 bytes from IOD #2 (peer unspecified) EID 26 GET / HTTP/1.0 NSOCK (4.4280s) Callback READ SUCCESS for EID 26 (peer unspecified) (15 bytes) NSOCK (4.4280s) Write request for 16 bytes to IOD #1 EID 35 [64.13.134.52:80] NSOCK (4.4280s) Callback: WRITE SUCCESS for EID 35 [64.13.134.52:80] NSOCK (4.4280s) Read request for 0 bytes from IOD #2 (peer unspecified) EID 42 For comparison, here is what Eric Jackson's nc (The nc available in Fedora 10's package repository) shows in verbose mode for the same connection: # nc -v scanme.nmap.org 80 Connection to scanme.nmap.org 80 port [tcp/http] succeeded! GET / HTTP/1.0 [David] o Final polishing of our GSoC pages. [Fyodor] o Advertise widely for Nmap GSoC applicants [Fyodor] o [Ncat] We should (maybe) consider a way for people to choose usernames in --chat. o Removing this for now. We can add it back if we decide we really want this. o Deal with new Python 2.6 Zenmap build warnings: C:\Python26\lib\site-packages\py2exe\build_exe.py:16: DeprecationWarning: the sets module is deprecated import sets http://sourceforge.net/tracker/index.php?func=detail&aid=2314799&group_id=15583&atid=115583 [Bug in py2exe, will probably be fixed with a new version of py2exe once it is released and we upgrade. This isn't causing us any major problem anyway.] o When I scan large groups of hosts with OS detection enabled, I get groups of warnings like: Insufficient responses for TCP sequencing (0), OS detection may be less accurate Insufficient responses for TCP sequencing (0), OS detection may be less accurate Insufficient responses for TCP sequencing (0), OS detection may be less accurate Insufficient responses for TCP sequencing (0), OS detection may be less accurate Insufficient responses for TCP sequencing (0), OS detection may be less accurate Note how it doesn't even tell the relevant IP address, and it isn't included in an individual host section. We should probably either include it in the section for an individual host, like we do with "OSScan results may be unreliable because we could not find at least 1 open and 1 closed port", or (not quite as good) include the relevant IP address in the error message. And we may or may not want to require verbose mode. o Ncat chat should bomine the "already connected" user list into one line, like: already connected: 69.232.238.42 is connected as , 206.81.65.43 as , 69.232.238.42 as o [Ndiff] Maybe Ndiff should display changes to version detection and OS detection information? [David] o Version detection done, now just needs OS detection. o When I start ncat chat with this tcsh command: ncat -l --chat scanme.nmap.org < /dev/null >& /dev/null & The first client to connect to the chat becomes user0 and doesn't work quite right. Messages user0 type get transmitted to other clients, but user0 does not see their messages. Nore does user0 get the normal connection announcement upon connecting. If I quit user0, the next client to connect becomes user0 again and has the same problem. If I start ncat on the server with "ncat -l --chat scanme.nmap.org" (no redirection), other clients can connect with no problems. o Ncat --chat should probably announce to everyone (including the new person) when someone connects. This tells the new person their username, and lets everyone else know about the new connection. [David] o We should also tell the new person (and possibly everyone on the channel) the list of existing participants. o SoC ideas page [Fyodor] o Nmap 4.85BETA4 release [Fyodor] o [Ncat] Wouldn't it be nice if we could support --exec (and maybe some sort of partial-emulated --sh-exec) on Windows? [David] o Almost working! We found some problems with "ncat.exe -v -l --sh-exec "ncat -v scanme.nmap.org" o [Ncat] Can we use it as an IPv4 <-> IPv6 gateway? If so (or if we can add it), it should be added to the ncat guide feature list. o Yes, David tried it with --sh-exec and it worked. o [Ncat] We should probably make it work without OpenSSL. When I try ./configure --without-openssl on latest svn Nmap, Ncat build fails with: gcc -MM -I../libpcap -DHAVE_CONFIG_H -D_FORTIFY_SOURCE=2 -I. -I.. -I../nsock/include/ -I../nbase ncat_main.c ncat_connect.c ncat_core.c ncat_listen.c ncat_proxy.c ncat_broker.c ncat_hostmatch.c ncat_ssl.c util.c sys_wrap.c > makefile.dep make[2]: Leaving directory `/mondo/fyodor/nmap/ncat' make[2]: Entering directory `/mondo/fyodor/nmap/ncat' gcc -I../libpcap -DHAVE_CONFIG_H -D_FORTIFY_SOURCE=2 -I. -I.. -I../nsock/include/ -I../nbase -c ncat_main.c -o ncat_main.o ncat_main.c: In function ‘main’: ncat_main.c:536: error: ‘struct options’ has no member named ‘ssl’ ncat_main.c: In function ‘ncat_listen_mode’: ncat_main.c:646: error: ‘struct options’ has no member named ‘ssl’ ncat_main.c:646: error: ‘struct options’ has no member named ‘sslcert’ ncat_main.c:646: error: ‘struct options’ has no member named ‘sslkey’ make[2]: *** [ncat_main.o] Error 1 make[2]: Leaving directory `/mondo/fyodor/nmap/ncat' make[1]: *** [build-ncat] Error 2 make[1]: Leaving directory `/mondo/fyodor/nmap' make: *** [static] Error 2 o [Ncat] Defensive coding review of Ncat --chat (talk) o [Ncat] As SSL server it should not crash when someone connects in w/o SSL and does ^C. When David tried it during our chat, the ncat servr "ncat --broker --ssl-key test-key.pem --ssl-cert test-cert.pem --ssl --chat -l" crashed with: SSL_accept(): error:00000000:lib(0):func(0):reason(0). Also, when a Windows SSL clients joined and then left, the server died with "Broken pipe o [Ncat] --chat should probably only allow reasonable chars, to avoid cntrl-chars, etc. o Nmap should treat ports named "unknown" in nmap-services the same way (from a naming perspective) as it treats ports which are not listed at all. See http://seclists.org/nmap-dev/2009/q1/0589.html. o Ncat user guide "Emulating Diagnostic Services" page has a very long UDP chargen server line which causes wrapping problems in web browsers (e.g. it widens the page substantially). It should probably be split into multiple lines. [David] o Ncat user guide proxying section says "The only exception is when listing a proxy host by IPv6 address; then the port is required." Why would we require a port number for IPv6 rather than just use the same defaults as we do for IPv4? [David explained that this is because to do otherwise would be ambiguous because IPv6 uses : for separaters, so we wouldn't know how to handle things like FF::10:80] o [Ncat] Perhaps we should make --ssl work in --chat. If nothing else, it might be useful if you want to reduce the number of people connecting with telnet, etc. rather than ncat. o [Ncat] --talk should probably be changed (in the code and documentation) to --chat, as Ncat chat has a much nicer ring to it, IMHO. --talk should remain as an alias to --chat, but we don't need to document it. [David] o Ncat Windows issue where you make a connection and then take several seconds to type in a line to the server, Ncat wrongly times out when trying to write your line to the remote server. [David] o Ncat write timeout problems cause client to quit due to write timeout sometimes. [David] Examples: o yes | ncat localhost o when we paste a few lines into the terminal window in an Ncat chat o Defensive coding review of ncat_proxy.* [David] o Process the latest version detection submissions. We now have more than 1,700 of them queued up. [Doug] o Write Ncat users' guide, demonstrating all the neat stuff you can do with it. This should probably be in DocBook XML so it can be an NNS chapter. You might want to query nmap-dev for list of neat things people do with ncat (or look around for what people do with nc). Testing it out for examples might expose areas for improvement as well. [David] o Look at Dario Ciccarone's email from 5/1/07 about IPID sequence issues, and consider adding IPID sequence test for closed-port-tcp as they apparently can be different. [David] o Also fix bug which causes SEQ to not be printed if the TCP open port tests fail to produce results, even though the II and (upcoming) CI tests may have useful results. [David] o NSE should offer some way to sleep/yield for a given amount of time. This would allow other scripts to run while a script has nothing to do. Possible uses: o Many services have rate limits (or you might just want to use them for politeness). For example, a web site spidering application might want to limit HTTP requests to some number per second to avoid pissing off the target webmaster more than is necessary (or prevent getting auto-blocked). Similarly, whois servers often will block IPs which query them too often in a short period. Or maybe you don't want to exceed the threshold limits of an IDS. o Example current scripts which might benefit: sql-injection, whois (possibly), pop3-brute, etc. o If we don't currently have a way for a cpu-bound NSE script to yield, then perhaps this could help us implement such a mechanism. But maybe coroutine.yield already does the trick. o The mechanism needs to be documented, and ideally should be implemented in at least one of the scripts shipped with Nmap. o Consider adding a way for requesting timing status updates at a given interval (such as every 5 seconds) to XML and/or normal output. This would be useful for people who run Nmap from scripts or other higher level applications. [David] o Ncat --allow/--deny bug: "--allow and --deny only support host specification by IP address, and give no warning when you use another form such as a host name." Should probably use same syntax as --exclude. We also want to at least do verification at the beginning to make sure all the entries are legitimately formed. We probably want to do things like DNS resolution at the beginning too. Otherwise we might have a DNS failure when we actually get a connection and perhaps have to reject the connection wrongly, or risk a false negative. [David] o Fix this overflow: Stats: 93:57:40 elapsed; 254868 hosts completed (2048 up), 2048 undergoing UDP Scan UDP Scan Timing: About 11.34% done; ETC: 03:21 (-688:-41:-48 remaining) [Done by David and Henri Doreau] o Ncat -- perhaps connection brokering should support UDP as well as (its existing support for) TCP? Actually this does raise issues such as deciding what list of UDP systems to forward a packet too. Its obviously not like TCP where you have a list of open connections. Ncat could build such a list, but, for example, would never know when to remove the host. For now, David is just going to adjust the error message to encourage people to email nmap-dev describing their usage scenario if they want this feature. o Ncat documentation should note that no SSL certificate verification is done (maybe we should offer an option to do so, if OpenSSL makes that easy). o Done in the new Ncat user's guide o Fix dns-zone-transfer infinite recursion bug described at http://seclists.org/nmap-dev/2009/q1/0317.html. It sounds like the best approach is to use our dns.lua library rather than having dns-zone-transfer do its own DNS packet parsing. o Fix XML escaping issue so that improper chars from NSE scripts or elsewhere can't cause corrupt XML files. See http://seclists.org/nmap-dev/2009/q1/0316.html for an example. [David] o Look into whether we should increase the frequency of port scan pings. See http://seclists.org/nmap-dev/2008/q1/0096.html . Note that Fyodor already increased them a bit in 2008. Might not need more. [David did extensive testing of this one already] o Find way to document NSE library script arguments and perhaps have them bubble up to scripts themselves. For example, I had to read the SNMP library source code to determine the script argument to specify the SNMP community name for snmp-sysdescr (https://nmap.org/nsedoc/scripts/snmp-sysdescr.html). Maybe we could just standardize on something like we do with SMB library and the scripts which call it (https://nmap.org/nsedoc/modules/smb.html, https://nmap.org/nsedoc/scripts/smb-check-vulns.html). [David] o If it wouldn't bloat things too much, it would be nice to include ndiff in the Nmap win32 zip distribution files. o Reported NSE crash: "Assertion failed - file ..\nse_main.cc line 314 lua_gettop(L_script_scan) == 0" o He says: "After looking at this closer, it appears the assertion occurs if I include the IP where the scan is run from. For us, I'm running this on IP 57, which is a VMware Windows Server image. If I eliminate that IP from the range it successfully completed the scan for all other devices." o Seems to be fixed. He can no longer reproduce the problem with 4.85BETA3. o Deal with GTK DLL problem with Nmap 4.85BETA1: [Fyodor] o David's installer seems to work--he's using a different GTK distribution. I'll try that. Works! Done! o Details on problem: http://seclists.org/nmap-dev/2009/q1/0207.html o Quick workaround done for 4.85BETA2, but better solution needed. o "SCRIPT ENGINE (250.600s): ./scripts/rpcinfo.nse against a.b.c.d: ended with error: ./nselib/datafiles.lua:114: attempt to index global 'arg' (a nil value)" -- http://seclists.org/nmap-dev/2009/q1/0227.html [Patrick] o Consider making the TODO list public o Done: http://seclists.org/nmap-dev/2009/q1/0175.html o Probably remove all of the "done" items since that is easier than reviewing them. o Might as well add to insecure.org/nmap/data/ o Maybe a bug tracker is a better approach. o [NPING] Fix compilation on Solaris. See http://seclists.org/nmap-dev/2010/q1/870.