Instructions for contributors

We are interested in various different kinds of improvement for scrapd; please feel free to raise an Issue if you would like to work on something major to ensure efficient collaboration and avoid duplicate effort.


Use the provided templates to file an Issue or a Pull Request.

Create a topic branch from where you want to base your work.

For formatting the files properly, please use YAPF.In the root directory of the project, run the following command:

make format

Make sure you added tests to validate your changes.

Run all the tests to ensure nothing else was accidentally broken.

Commit messages must start with a capitalized and short summary (max. 50 chars) written in the imperative, followed by an optional, more detailed explanatory text which is separated from the summary by an empty line.

Commit messages should follow best practices, including explaining the context of the problem and how it was solved, including in caveats or follow up changes required. They should tell the story of the change and provide readers understanding of what led to it. Please refer to How to Write a Git Commit Message for more details.

Commit example

Use Docker Hub build environment values

Uses the Docker Hub build environment values in order to ensure the
correct version of ScrAPD is installed into the image.

Fixes scrapd/scrapd#54

The following commit is a good example as:

  1. The fist line is a short description and starts with an imperative verb
  2. The first paragraph describes why this commit may be useful
  3. The last line points to an existing issue and will automatically close it.

Formatting your code

There is also a lot of YAPF plugins available for different editors. Here are a few:

Developer setup

Setup a local dev environment:

make venv
source venv/bin/activate

Run the CI tasks locally

make ci

Use make help to see all the available make targets.


How to test the regexes

cd tests/data
curl -sLO
export FIELD="Date:"
grep -h -e "${FIELD}" -C 1 traffic-fatality-* | pbcopy

Paste the result there, choose python, and work on your regex.