rastrea2r¶

Ever wanted to turn your AV console into an Incident Response & Threat Hunting machine? Rastrea2r (pronounced “rastreador” - hunter- in Spanish) is a multi-platform open source tool that allows incident responders and SOC analysts to triage suspect systems and hunt for Indicators of Compromise (IOCs) across thousands of endpoints in minutes. To parse and collect artifacts of interest from remote systems (including memory dumps), rastrea2r can execute sysinternal, system commands and other 3rd party tools across multiples endpoints, saving the output to a centralized share for automated or manual analysis. By using a client/server RESTful API, rastrea2r can also hunt for IOCs on disk and memory across multiple systems using YARA rules. As a command line tool, rastrea2r can be easily integrated within McAfee ePO, as well as other AV consoles and orchestration tools, allowing incident responders and SOC analysts to collect forensic evidence and hunt for IOCs without the need for an additional agent, with ‘gusto’ and style!

User Guide¶

This section of the documentation provides user focused information such as installing and quickly using this package.

Description¶

Ever wanted to turn your AV console into an Incident Response & Threat Hunting machine? Rastrea2r (pronounced “rastreador” - hunter- in Spanish) is a multi-platform open source tool that allows incident responders and SOC analysts to triage suspect systems and hunt for Indicators of Compromise (IOCs) across thousands of endpoints in minutes. To parse and collect artifacts of interest from remote systems (including memory dumps), rastrea2r can execute sysinternal, system commands and other 3rd party tools (including the custom scripts) across multiples endpoints, saving the output to a centralized share for automated or manual analysis. By using a client/server RESTful API, rastrea2r can also hunt for IOCs on disk and memory across multiple systems using YARA rules. As a command line tool, rastrea2r can be easily integrated within McAfee ePO, as well as other AV consoles and orchestration tools, allowing incident responders and SOC analysts to collect forensic evidence and hunt for IOCs without the need for an additional agent, with ‘gusto’ and style!

Current functionality Supported: * Fast Triaging: Execute Sysinternals tools, or any other 3rd party batch scripts (including custom scripts) to perform basic triaging ** Windows Only * Forensic Artifact Collection: Capabilities to Create snapshots quickly (Implements a wrapper for CyLR tool, which collects forensic artifacts from hosts with NTFS file systems quickly, securely and minimizes impact to the host.) **Windows Only * Web History: Collect the Browser History (Currently supports IE, Chrome, Firefox only) * Prefetch Tool: Collect the prefetch data in Windows as they are great artifacts for forensic investigations to analyze applications that have been run on a system. ** Windows only * Memory Dump: Acquires a memory dump from the endpoint ** Windows only * Yara Disk: Yara scan for file/directory objects on disk * Yara Mem: Yara scan for running processes in memory

Rastrea2r now also supports pushing the Scan Results to a Restful Server using HTTP. This functionality allows the users to deploy rastrea2r on their enterprises so that they can execute different rastrea2r commands to collect and triage the data and later store the Yara disk or Yara Mem results onto the Server for further analysis.

A high level design of the rastrea2r deployment on end points can be seen as below:

Rastrea2r allows users to specify the list of commands or batch scripts to be executed during “triage” via a configuration file, which is located here_. .._here: src/rastrea2r/rastrea2r.ini

Notes¶

For memdump and triage modules, SMB shares must be set up in this specific way:

Binaries (sysinternals, batch files and others) must be located in a shared folder called TOOLS (read only)

\path-to-share-foldertools
Output is sent to a shared folder called DATA (write only)

\path-to-share-folderdata
For yara-mem and yara-disk scans, the yara rules must be in the same directory where the server is executed from.

Dependencies¶

yara-python==3.7.0
psutil==5.4.6
Requests=2.19.1
Pyinstaller=3.3.1

Quickstart¶

Clone the project to your local directory (or download the zip file of the project)

$git clone https://github.com/rastrea2r/rastrea2r.git
$cd rastrea2r

All the dependencies necessary for the tool to run can be installed within a virtual environment via the provided makefile.

$make help
help                           - display this makefile's help information
venv                           - create a virtual environment for development
clean                          - clean all files using .gitignore rules
scrub                          - clean all files, even untracked files
test                           - run tests
test-verbose                   - run tests [verbosely]
check-coverage                 - perform test coverage checks
check-style                    - perform pep8 check
fix-style                      - perform check with autopep8 fixes
docs                           - generate project documentation
check-docs                     - quick check docs consistency
serve-docs                     - serve project html documentation
dist                           - create a wheel distribution package
dist-test                      - test a wheel distribution package
dist-upload                    - upload a wheel distribution package

Create a virtual environment with all dependencies

$make venv
//Upon successful creation of the virtualenvironment, enter the virtualenvironment as instructed, for ex:
$source /Users/ssbhat/.venvs/rastrea2r/bin/activate

Start the rastrea2r server by referring to: https://rastrea2r-server.readthedocs.io/en/latest/?badge=latest
Now execute the client program, depending on which platform you are trying to scan choose the target python script appropriately. Currently Windows, Linux and Mac platforms are supported.

Note

Following instructions explain the steps on a Windows, but on Mac and linux the steps should follow the same except that you would execute the client from the specified platform folder. On Windows PC’s, make file system is not supported and if you need to execute rastrea2r client then you need to create the virtualenvironment manually and install the dependencies on it using pip install -r requirements.txt.

$python rastrea2r_windows.py -h
usage: rastrea2r_windows.py [-h] [-v] {yara-disk,yara-mem,triage,memdump,triage,web-hist,prefetch} ...

Rastrea2r RESTful remote Yara/Triage tool for Incident Responders

positional arguments:  {yara-disk,yara-mem,triage,memdump,triage,web-hist,prefetch}

modes of operation
 yara-disk           Yara scan for file/directory objects on disk
 yara-mem            Yara scan for running processes in memory
 memdump             Acquires a memory dump from the endpoint
 triage              Collect triage information from endpoint
 web-hist            Generates web history for specified user account
 prefetch            Generates prefetch view
 collect             Acquires artifacts from the endpoint


optional arguments:
 -h, --help            show this help message and exit
 -v, --version         show program's version number and exit


Further more, the available options under each command can be viewed by executing the help option. i,e

$python rastrea2r_windows.py yara-disk -h
usage: rastrea2r_osx.py yara-disk [-h] [-s] path server rule

positional arguments:
path          File or directory path to scan
server        rastrea2r REST server
rule          Yara rule on REST server

optional arguments:
-h, --help    show this help message and exit
-s, --silent  Suppresses standard output

For ex, on a Mac system you would do:

$cd src/rastrea2r/windows/

$python rastrea2r_windows.py yara-disk /opt http://localhost example.yara

Executing rastrea2r.exe on Windows¶

On windows platform rastrea2r can be compiled into a single exe file so that it can later be executed on the endpoints. To do so, execute the build_exe.bat file located under “src” folder so that it can generate the exe from the latest codebase. Upon successful execution it will generate the exe under dist folder. And all the functionalities of rastrea2r described above can be executed with exe as well. for ex:

$rastrea2r.exe collect tools.myserver.com data.myserver.com

//where tools.myserver.com has a shared smb folder called "tools" with readonly access as all the tools specified in the rastrea2r.ini must be present in this tools folder prior to executing the commands
//      data.myserver.com is server which has shared folder called "data" with write access so that all the results from the rastrea2r command can be placed here.

Note

In order to generate the rastrea2r.exe the build_exe.bat internally uses pyinstaller spec file. Which would have a dependency on the Windows C++ binaries which must be installed on the machine prior to calling the build_exe.bat. Follow these instructions to update the spec file.

1. Download and install this Windows SDK: https://developer.microsoft.com/en-us/windows/downloads/windows-10-sdk
2. These dlls can then be found here (or Program Files x86):
    C:\Program Files\Windows Kits\10\Redist\ucrt\DLLs
3. Update your pathex variable in your spec file to tell pyinstaller to look there:

    pathex=['C:\\Users\\grey_hat\\Desktop\\csm\\test',
        'C:\\Program Files (x86)\\Windows Kits\\10\\Redist\\ucrt\\DLLs\\x86',
        'C:\\Program Files (x86)\\Windows Kits\\10\\Redist\\ucrt\\DLLs\\x64']

Report Bugs¶

Report bugs at the issue tracker.

Please include:

Operating system name and version.

Any details about your local setup that might be helpful in troubleshooting.

Detailed steps to reproduce the bug.

Contributing to rastrea2r project¶

The Developer Documentation provides complete information on how to contribute to rastrea2r project

Demo videos on Youtube¶

Video 1: Incident Response / Triage with rastrea2r on the command line - https://youtu.be/uFIZxqWeSyQ
Video 2: Remote Yara scans with rastrea2r on the command line - https://youtu.be/cnY1yEslirw
Video 3: Using rastrea2r with McAfee ePO - Client Tasks & Execution - https://youtu.be/jB17uLtu45Y

Presentations¶

rastrea2r at BlackHat Arsenal 2016 (check PDF for documentation on usage and examples) https://www.blackhat.com/us-16/arsenal.html#rastrea2r

https://github.com/aboutsecurity/Talks-and-Presentations/blob/master/Ismael_Valenzuela-Hunting_for_IOCs_rastrea2r-BH_Arsenal_2016.pdf
Recording of talk on rastrea2r at the SANS Threat Hunting Summit 2016

https://www.youtube.com/watch?v=0PvBsL6KKfA&feature=youtu.be&a

Credits & References¶

To Robert Gresham Jr. (@rwgresham) and Ryan O’Connor (@_remixed) for their contributions to the Triage module. Thanks folks!
To Ricardo Dias for the idea of using a REST server and his great paper on how to use Python and Yara with McAfee ePO: http://www.sans.org/reading-room/whitepapers/forensics/intelligence-driven-incident-response-yara-35542

Developers Guide¶

Contributing Guide¶

Contributions are welcome and greatly appreciated!

Workflow¶

A bug-fix or enhancement is delivered using a pull request. A good pull request should cover one bug-fix or enhancement feature. This ensures the change set is easier to review and less likely to need major re-work or even be rejected.

The workflow that developers typically use to fix a bug or add enhancements is as follows.

Fork the rastrea2r repo into your account.

Obtain the source by cloning it onto your development machine.

$ git clone git@github.com:your_name_here/rastrea2r.git
$ cd rastrea2r

Create a branch for local development:
```
$ git checkout -b name-of-your-bugfix-or-feature
```
Now you can make your changes locally.

Note

Below instructions assumes that you are developing on a Mac or Unix system. If you are on Windows machine, then please note that you have to have the MakeFile support is installed. One way to install the make files is by using the mingw. Please refer to: http://www.mingw.org/wiki/getting_started for more details

Familiarize yourself with the developer convenience rules in the Makefile.
```
$ make help
```
Create and activate a Python virtual environment for local development.
```
$ make venv
$ source path/to/<venv-name>/bin/activate
(venv) $
```
The rule creates the virtual environment outside the project directory so that it never accidentally gets added to the change set.

Note

(venv) is used to indicate when the commands should be run within the virtual environment containing the development dependencies.
Develop fix or enhancement:
- Make a fix or enhancement (e.g. modify a class, method, function, module, etc).
- Update an existing unit test or create a new unit test module to verify the change works as expected.
- Run the test suite.
```
(venv) $ make test
```
  See the Testing section for more information on testing.
- Check code coverage of the area of code being modified.
```
(venv) $ make check-coverage
```
  Review the output produced in docs/source/coverage/coverage.html. Add additional test steps, where practical, to improve coverage.
- The change should be style compliant. Perform style check.
```
(venv) $ make check-style
```
  See the Code Style section for more information.
- The change should include type annotations where appropriate. Perform type annotations check.
```
(venv) $ make check-types
```
  See the Type Annotations section for more information.
- Fix any errors or regressions.
The docs and the change log should be updated for anything but trivial bug fixes. Perform docs check.
(venv) $ make docs
See the Documentation section for more information.
Commit and push changes to your fork.
```
$ git add .
$ git commit -m "A detailed description of the changes."
$ git push origin name-of-your-bugfix-or-feature
```
A pull request should preferably only have one commit upon the current master HEAD, (via rebases and squash).
Submit a pull request through the service website (e.g. Github, Gitlab).
Check automated continuous integration steps all pass. Fix any problems if necessary and update the pull request.

Testing¶

The rastrea2r project implements a regression test suite that improves developer productivity by identifying capability regressions early.

Developers implementing fixes or enhancements must ensure that they have not broken existing functionality. The rastrea2r project provides some convenience tools so this testing step can be quickly performed.

Use the Makefile convenience rules to run the tests.

(venv) $ make test

To run tests verbosely use:

(venv) $ make test-verbose

Alternatively, you may want to run the tests suite directly. The following steps assume you are running in a virtual environment in which the rastrea2r package has been installed. If this is not the case then you will likely need to set the PYTHONPATH environment variable so that the rastrea2r package can be found.

(venv) $ cd tests
(venv) $ python -m unittest

Individual unit tests can be run also.

(venv) $ python -m test_basic

Coverage¶

The coverage tool can be run to collect code test coverage metrics.

Use the Makefile convenience rule to run the tests.

(venv) $ make check-coverage

The test code coverage report can be found here

Code Style¶

Adopting a consistent code style assists with maintenance.

Use the Makefile convenience rule to check code style compliance.

(venv) $ make check-style

A separate style fix rule is available to automate fixing minor problems. More complicated problems will need to be fixed manually.

(venv) $ make fix-style

Type Annotations¶

The code base contains type annotations to provide helpful type information that can improve code maintenance.

Use the Makefile convenience rule to check no issues are reported.

(venv) $ make check-types

Documentation¶

To rebuild this project’s documentation, developers should use the Makefile in the top level directory. It performs a number of steps to create a new set of sphinx html content.

(venv) $ make docs

To quickly check consistency of ReStructuredText files use the dummy run which does not actually generate HTML content.

(venv) $ make check-docs

To quickly view the HTML rendered docs, start a simple web server and open a browser to http://127.0.0.1:8000/.

(venv) $ make serve-docs

Release Process¶

The following steps are used to make a new software release.

The steps assume they are executed from within a development virtual environment.

Check that the package version label in __init__.py is correct.

Create and push a repo tag to Github. As a convention use the package version number (e.g. YY.MM.MICRO) as the tag.

$ git checkout master
$ git tag YY.MM.MICRO -m "A meaningful release tag comment"
$ git tag  # check release tag is in list
$ git push --tags origin master

This will trigger Github to create a release at:

https://github.com/{username}/rastrea2r/releases/{tag}

Create the release distribution. This project produces an artefact called a pure Python wheel. The wheel file will be created in the dist directory.
```
(venv) $ make dist
```
Test the release distribution. This involves creating a virtual environment, installing the distribution into it and running project tests against the installed distribution. These steps have been captured for convenience in a Makefile rule.
```
(venv) $ make dist-test
```
Upload the release to PyPI using
```
(venv) $ make dist-upload
```
The package should now be available at https://pypi.org/project/rastrea2r/