Every now and then I see someone struggling with http requests that do not produce the desired result. People start to look into server logs, but mostly they strip POST bodies or pseudonomize GET requests.
Sometimes you don’t even have access to the server logs at all. Struggle no more, mitmproxy to the rescue.

First of all we need to imagine how a proxy setup will look like.
The proxy acts as a facilitator between Client and Server. It will also log the request and responses. Some proxies can also modify HTTP header to enable or disable certain features.

In this example we will use a docker setup with mitmproxy

The docker-compose.yml will just map the ports and for mitmproxy enable the webinterface so we can see the requests:

version: '3'
services:
  mitmproxy:
    image: mitmproxy/mitmproxy
    command: mitmweb --web-host 0.0.0.0
    restart: unless-stopped
    ports:
      - 8080:8080
      - 8081:8081

To start this setup, issue a docker-compose up and docker will do its magic. To test if the mitmproxy has started succesfully, open http://localhost:8081 to see the dashboard.

Python

Now let us create some minimal Python code to talk to the echo server:

import requests

req = requests.get('http://scooterlabs.com/echo?foo=bar')
print(req.text)

You will see the response from the echo server, but how can you see what exactly python was sending to the server and how exaclty did the response from the server look like?

Now this is where we need to configure the proxy:

import requests

proxies = {
    'http': 'http://localhost:8080',
}

req = requests.get('http://scooterlabs.com/echo?foo=bar', proxies=proxies)
print(req.text)

Run the modified script and you will see mitmproxy showing all the missing information:

Node.js

With node the setup is similiar, I use plain node, there are other ways with other packages.

const http = require('http');

const options = {
    host: 'scooterlabs.com',
    port: '80',
    path: '/echo',
}

const request = http.request(options, (response) => {
    let body = '';
    response.on('data', (data) => {
        body += data;
    });

    response.on('end', (end) => {
        console.log(body);
    });
});
request.end();

This will log the echo response to your console, as expected. Since there is no native option for proxy, you could modify the options like this:

const http = require('http');

const options = {
    host: 'localhost',
    port: '8080',
    path: 'http://scooterlabs.com/echo',
}

const request = http.request(options, (response) => {
    let body = '';
    response.on('data', (data) => {
        body += data;
    });

    response.on('end', (end) => {
        console.log(body);
    });
});
request.end();