Using foorep part 2 - CLI

foorep is your own personal repository for forensic artifacts and malware samples.

In part one we went through installing foorep. In this part I will introduce you to using the command line interface. If you want an introduction to what foorep is and why I created it, please read this post.

You can interact with foorep in several ways. You can use the command line interface, click in the built in webUI or integrate it directly into your own python programs. 

The interface is designed to have sub-commands for different actions

add

$ foorep add /path/to/sample

This command adds a sample to the repository. The system runs all the available plugins to gather more information about the file. For example, if it is a PE file the pefile-plugin will run and if it is a image with EXIF data in it, the exif-plugin will parse it and add the info to the database and so on.

remove

$ foorep remove <id>

This command removes a sample from the repository.

search

$ foorep search <query>

This command searches the repository for matching samples. Today you can search after md5,sha1,sha256,sha512 and filename. The search functionality will get more sophisticated.

dump

$ foorep dump <id>

This command will dump the raw JSON document to stdout. Used mostly for debugging

annotate

$ foorep annotate -t <type> -m <content>

Annotations is a big deal in foorep. It is here that you add value and context to your samples. For example, both comments and tags is simple annotations. All the results from the plugins is saved as annotations in the database.

 

list

$ foorep list -l 10

This command list the samples in the database. Add -l to limit the output, or use -l 0 to list all.

I hope that this will get you going using foorep. Next time we will look at using the built in web interface. Here is a preview:

Using foorep part 1 - Installing

foorep, your personal forensic repository. In this article I will show you how to install foorep on different versions of Ubuntu and on SIFT Workstation.
The second post in this series will be all about using the CLI interface!

For an overview of what foorep is and why I created it, take a look at my earlier blogpost.

Ubuntu 12.04 and 12.10

On Ubuntu 12.04 and 12.10 all the dependencies are available from the apt sources. In 3 simple steps, you will be up and running.

1. Install mongodb
$ sudo apt-get install mongodb
2. Install support for parsing exif
$ sudo apt-get install python-pyexiv2
3. Install foorep
$ sudo pip install foorep



Ubuntu 10.04 and SIFT Workstation 2.14 (Ubuntu 9.10)

The provided version of mongodb in older Ubuntu is lacking some features that we depend on, so we need to install mongodb from 10gens repository. Also the package python-magic is too old. We fix all this with some additional steps.

1. Uninstall old mongodb
$ sudo apt-get remove mongodb
2. Add 10gen Ubuntu repository to sources.list
$ sudo echo "deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen" >> /etc/apt/sources.list
3. Add 10gen key
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10
4. Update apt
$ sudo apt-get update
5. Install mongodb
$ sudo apt-get install mongodb-10gen
6. Install support for parsing exif
$ sudo apt-get install python-pyexiv2
7. Remove python-magic, the version bundled with Ubuntu is too old.
$ sudo apt-get remove python-magic
8. Install foorep
$ sudo pip install foorep



Play

After you have installed foorep, you are ready to fill it with content. I will write about this in the next post, but for the inpatient:

• foorep add /pat/to/malware
• foorep list
• foorep dump id
• foorepd (start the built in webserver)
• Browse to http://127.0.0.1:4780

Happy hacking!

Forensic repository for both humans and machines

foorep is an open source forensics/malware repository written in Python that is easy to install, scales if you need it and provides interfaces for both humans and machines.

The "Hello World" for #dfir people today seems to be creating their own malware repository. I needed to learn mongoDB and thought that yet another foo repository couldn't hurt.. So after a couple of late nights, we have this:

Features at a glance

• CLI
• WebUI
• RESTful API
• Python library
• Search
• Simple but powerful annotations
• Plugin system for static analysis

Easy to install
If you are using Ubuntu it is a 2-step experience (MacOSX is untested, but will probably fail. I will fix this). First you need a mongodb database. If you have one running you can use that, if you don't you need to install one. By default mongodb will listen on localhost with no authentication. If this is something you can live with, you are good to go.

Note: This is a _very_ early release/hack done on my non-existing spare time, so there are rough edges all over. Drop me a line if something isn't working for you or even better, send a bug report!

1) $ apt-get install mongodb
2) $ pip install foorep

Then try it out:
$ foorep add /path/to/malware
$ foorep list

Or start the built in webserver and use the webUI
$ foorepd

Then open your browser and go to http://localhost:4780

Annotations
foorep has a very flexible way to add annotations to your samples. You are able to tag and comment, but it doesn't stop there. Annotations in foorep are simple JSON objects that we embed in the metadata document. This makes it possible to invent new types of annotations as we go.

Plugins
In order to do something more than just store the files, I created a simple plugin system that automatically do some static analysis on the malware being uploaded. The plugin will recieve a filehandler or a path to a file and it should return a foorep annotation (python dictionary).
Implemented plugins:

• PEfile - if the file is a PE file, pick it apart and do some analysis
• exif - if the file is a image with exif data in it, annotate the sample with the exif tags (currantly you need to install python-exiv2, apt-get install python-pyexiv2)

I am planning on creating plugins for Yara, Virustotal or Team Cymru Malware Repository to name a few.

Sharing
This is something that will be implemented in the future. But I will integrate fordrop in foorep to make it possible to share and federate the samples in your repository with others. Making it a decentralized and distributed malware repository. We will use XMPP pubsub for this. See fordrop for more information.

The use of mongoDB

MongoDB (from "humongous") is a scalable, high-performance, open source NoSQL database. 

MongoDB is a document oriented database were you store JSON documents. This is a perfect fit for a malware repository, and if I can do without SQL schemas I'm happy!
foorep stores metadata from malware samples as JSON documents, making them searchable. We also store the raw files in mongoDB, using GridFS.

Use in other python programs

import foorep
my_repo = foorep.Repository()
my_repo.insert('/path/to/malware')



RESTful API
If you are a machine and want to consume and produce data in foorep but you are not a python, you can use the RESTful HTTP based API. In this initial release the API is just read-only, but I plan on implementing the rest in the next release. I will write more on this soon.

curl http://127.0.0.1:4780/api/v1/file/d02aa8cb63084b6cbdfab9cb68343cf7

Hacking

Implemented in python with some test coverage (adding more tests is a priority as always). If you would like to help out adding features or crushing bugs the source is available over at github.

Open source

This project is released as open source using a 2-clause BSD license. This basically means that you can take it and do whatever you like with it. Please do.