# HTTP Toolkit HTTP Toolkit This is one page of public article previews, not the complete archive. Follow Next page to continue. Summaries are not the original full articles. ## Dictionary compression for web data: how it works and how to use it with Node.js DevFeed: [Dictionary compression for web data: how it works and how to use it with Node.js]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2026-02-23T13:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [Compression](), [Algorithm](), [JavaScript](), [Node.js](), [API](), [WebAssembly](), [JSON]() Tags: [algorithm](), [api](), [browser](), [browsers](), [compression](), [data](), [http](), [javascript](), [json](), [node-js](), [performance](), [standards](), [webassembly]() ### AI overview This tutorial explains dictionary compression, where the compressor and decompressor share known data so compressed output can reference it directly. It discusses using previous responses or custom dictionaries to reduce the size of JavaScript bundles, WebAssembly files, and structured API responses, and demonstrates the technique with Node.js. ### Source excerpt Dictionary compression could completely change how applications send data over the web. It's recently gained broad support, and offers absurd real-world traffic reductions: initial testing shows YouTube JS download size for returning desktop users shrinking up to 90% (!!!) compared to existing best-practice compression, while the Google search results HTML (arguably the most optimized content on the internet) shrinks nearly 50%. This works by initializing the (de)compression algorithm with a dictionary of data known in advance to both compressor & decompressor, so that the compressed data can just be references to that directly ("insert bytes 1 - 10,000 from the dictionary") without having to include the original data at all. This is applicable in a surprising number of scenarios, because most data we send (especially on the web) isn't completely novel or unpredictable. Today's JavaScript bundle shares 99% of its content with yesterday's JavaScript bundle - if the browser already has the old one, using that as a dictionary means you can compress down to (approximately) just the differences. This can work either using a previous response as the dictionary for the next response, or using an explicit custom dictionary; for many kinds of dynamic response, you do know large chunks of the data in advance, like all the keys in your API's JSON response, and many common values that might be included, and you can generate & preload a dictionary defining exactly this to efficiently cover those. In either case, this can drastically shrink JS bundles, WebAssembly files, known-structure API responses, or many other kinds of incrementally updated & diffable content - a lot of the worst offenders for bandwidth usage that have become very common on the modern web. This is now widely usable, safe to deploy without compatibility concerns, and surprisingly easy to set up. Here's a quick low-level demo for Node.js (v24.6+ or v22.19+) so you can play with the raw compression directly for ## Funding the OSS Stack: HTTP Toolkit & Open Source in 2025 DevFeed: [Funding the OSS Stack: HTTP Toolkit & Open Source in 2025]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2026-02-21T18:00:00Z Content type: opinion Language: en Sources: [HTTP Toolkit]() Topics: [Open Source](), [Maintainers]() Tags: [2025](), [announcements](), [business](), [funding](), [maintainers](), [open-source]() ### AI overview HTTP Toolkit describes its 2025 open-source funding, reporting $7,820 paid to maintainers and a cumulative total exceeding $20,000. The article lists recipients including Frida, Node.js, Electron, Mocha, Tauri, Servo, and Styled Components, and discusses broader support for open-source projects. ### Source excerpt HTTP Toolkit, like effectively all software businesses, depends on a huge quantity of open-source code for much of its fundamental functionality & infrastructure. Most of this is tirelessly maintained by volunteers, completely for free! This honestly is a great deal for the businesses, but it would be even better if these maintainers were actually rewarded for their hard work. As part of HTTP Toolkit's commitment to giving back to open source under the Open Source Pledge, a substantial chunk of revenue goes back into these open-source projects, to keep them healthy, reward maintainers for their tireless efforts, and encourage development on projects I care about to keep them moving forwards. What does that mean in practice? In 2025, httptoolkit paid $7,820 to open-source maintainers, taking the project's total past $20,000! Not bad for one person. $1200 to Frida $1100 to Node.js $320 to Electron $320 to Mobx $240 to blakeembrey $240 to szmarczak $240 to lpinca $240 to apocas $240 to anonrig $240 to Phosphor Icons $240 to mmaietta $240 to Regolith $240 to endoflife.date $240 to react-window $240 to fast-xml-parser $240 to node-datachannel $240 to Ajv $240 to Mocha $220 to louislam $220 to openapi-directory $200 to johnnyreilly $180 to Node.js Mobile $180 to Open Web Docs $160 to Tauri $140 to Servo $140 to Styled Components $40 to roderickvd $40 to Waydroid Open source has done a lot to power my tech education & career, and literally powers the foundations of HTTP Toolkit, so I'm more than happy to be able to support the upstream projects those depend on. None of these are earth-shaking amounts individually but cumulatively they add up, especially as other people and organizations add their own contributions in turn: Open Source Pledge companies are now collectively donating nearly $3 million a year in total back to maintainers! If contributions like the above were expected behaviour from everybody building on open-source work, OSS would be a very different place. Of ## HTTP/3 adoption is widespread in browsers and CDNs but limited in open-source tools DevFeed: [HTTP/3 adoption is widespread in browsers and CDNs but limited in open-source tools]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2025-03-12T16:00:00Z Content type: opinion Language: en Sources: [HTTP Toolkit]() Topics: [HTTP3](), [networking](), [Protocol (disambiguation)](), [Open Source]() Tags: [browsers](), [http](), [http-3](), [http3](), [networking](), [open-source](), [quic](), [standards]() ### AI overview This article examines the contrast between HTTP/3's widespread browser and CDN adoption and its limited support in major programming-language libraries, servers, Android networking, and Kubernetes tooling. It argues that HTTP/3 rollout beyond browsers and large CDNs remains at an early stage. ### Source excerpt HTTP/3 has been in development since at least 2016, while QUIC (the protocol beneath it) was first introduced by Google way back in 2013. Both are now standardized, supported in 95% of users' browsers, already used in 32% of HTTP requests to Cloudflare, and support is advertised by 35% of websites (through alt-svc or DNS) in the HTTP Archive dataset. We've developed a totally new version of HTTP, and we're on track to migrate more than 1/3 of web traffic to it already! This is astonishing progress. At the same time, neither QUIC nor HTTP/3 are included in the standard libraries of any major languages including Node.js, Go, Rust, Python or Ruby. Curl recently gained support but it's experimental and disabled in most distributions. There are a rare few external libraries for some languages, but all are experimental and/or independent of other core networking APIs. Despite mobile networking being a key use case for HTTP/3, Android's most popular HTTP library has no support. Popular servers like Nginx have only experimental support, disabled by default, Apache has no support or published plan for support, and Ingress-Nginx (arguably the most popular Kubernetes reverse proxy) has dropped all plans for HTTP/3 support punting everything to a totally new (as yet unreleased) successor project instead. Really it's hard to point to any popular open-source tools that fully support HTTP/3: rollout has barely even started. This seems contradictory. What's going on? I'm going to assume a basic familiarity with the differences between HTTP/1.1 (et al), HTTP/2 and HTTP/3 here. If you're looking for more context, http2-explained and http3-explained from Daniel Stenberg (founder & lead developer of curl) is an excellent guide. Why do we need more than HTTP/1.1? Let's step back briefly. Why does this matter? Who cares about whether HTTP/3 is being rolled out successfully or not? If browser traffic and the big CDNs support HTTP/3, do we even need it in other client or server implementat ## HTTP Toolkit is joining the Open Source Pledge DevFeed: [HTTP Toolkit is joining the Open Source Pledge]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2024-12-31T15:00:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [Open Source](), [Maintainers](), [Software]() Tags: [announcements](), [business](), [dependencies](), [funding](), [maintainers](), [open-source](), [open-source-software](), [security]() ### AI overview HTTP Toolkit announces that it is joining the Open Source Pledge, committing to publicly fund the maintainers of the open-source software it depends on at a minimum of $2,000 per full-time developer annually. ### Source excerpt The Open Source Pledge is a new push to make companies commit to funding the maintainers of the open-source software they depend on, and to publicly recognize the ones that do. HTTP Toolkit has donated back to maintainers for a few years now, but joining the Open Source Pledge today means formally committing to that, and to doing so publicly with a sustainable minimum level ($2000 per full-time developer, or higher) indefinitely into the future. What is the Open Source Pledge? HTTP Toolkit (and effectively 100% of other software businesses) depends on a huge quantity of open-source code for much of its fundamental functionality & infrastructure. Most of this is tirelessly maintained by volunteers, completely for free. Astonishingly, that basically works, and we've built an entire software industry on top of it. But it's not a fair deal, and over the years it has become increasingly clear that businesses replying on people maintaining their critical dependencies for free is not a good or sustainable approach (for either the businesses or the maintainers' mental health). Plenty of important open-source projects have been abandoned as maintainers moved on with their lives, while some have been actively removed entirely or even replaced with malicious content. For businesses these are big problems, but for maintainers it's not reasonable to expect that they'll continue actively supporting their projects for free indefinitely just because your business chose to use it. There have been many attempts to fix this, from the organizational (internally forking key dependencies, avoiding dependencies entirely where possible) to the purely technical (dependency locking & mirroring, security scanning of new releases). At the end of the day though, the only real solution to open source sustainability is to fund the maintainers you depend on. The Open Source Pledge aims to commit businesses to doing this, and to build a wider culture where that becomes the norm. The actual mechanis ## ERR\_PROXY\_CONNECTION\_FAILED errors with HTTP proxies DevFeed: [ERR\_PROXY\_CONNECTION\_FAILED errors with HTTP proxies]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2024-12-11T17:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [HTTP](), [browser](), [Security](), [debug](), [TLS (Transport Layer Security)](), [cURL](), [Postman]() Tags: [authentication](), [browser](), [chrome](), [curl](), [debugging](), [errors](), [http](), [issue](), [port](), [postman](), [protection](), [proxy](), [server](), [tcp](), [tls](), [tools]() ### AI overview This tutorial explains how to diagnose ERR_PROXY_CONNECTION_FAILED when using a local debugging proxy with Chrome or similar applications. It covers incorrect connection details, unreachable proxies, and antivirus software that intercepts local proxy traffic and rejects the proxy's TLS certificate. ### Source excerpt If you're using a local debugging proxy tool like HTTP Toolkit, you might run into the dreaded ERR_PROXY_CONNECTION_FAILED error in Chrome and other similar apps. This can be a very frustrating and unhelpful error! There's only a few possible causes though, and it's usually easy to fix. The Simple Case The simplest explanation is exactly what it says: the browser can't connect to your proxy. In the simple case this may be caused by a basic connection issue: you have the address, port or some authentication details wrong, or the details are correct but the proxy is just not reachable on your connection, and so the browser can't talk to it. The easiest way to confirm this is to try connecting to the details directly using an HTTP client tool like curl, Postman, or the HTTP Toolkit Send page, to manually check if the server is listening on the port you expect. If you see a TCP error then the details are wrong, but if you see any kind of HTTP error or similar then you know that the server does exist and it's reachable with the given details. If the failure comes from a browser intercepted by a tool like HTTP Toolkit though, where intercepted clients are preconfigured & launched for you, this is not what's happening - the server is definitely reachable (it's running locally) and the config is correct (it's been done automatically). Instead, this might be caused by something more complicated... Beyond the Simple Case If you know the settings are correct, and you know the proxy is reachable, what could cause this ERR_PROXY_CONNECTION_FAILED error? Once you're ruled out basic connection issues, the most likely cause is antivirus software on your computer that is intercepting your connections. This is becoming a common feature of some antivirus & security software, most notably ESET. It's not directly related to viruses at all, but is a separate security feature that's often enabled automatically for additional protection. What happens in this case is that your browser is tryi ## Designing API Errors DevFeed: [Designing API Errors]() Original publisher: [Read original article]() Author: HTTP Toolkit; Phil Sturgeon Published: 2024-09-09T15:30:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [API](), [HTTP]() Tags: [api](), [apis](), [developers](), [errors](), [http](), [standards]() ### AI overview The article explains how to design API errors by combining HTTP status codes with human-readable descriptions. It shows that status codes communicate broad categories and expected handling, while additional details help developers and users distinguish specific problems. ### Source excerpt When everything goes smoothly with an API, life is pretty straightforward: you request a resource, and voilà, you get it. You trigger a procedure, and the API politely informs you it's all gone to plan. But what happens when something goes pear-shaped? Well, that's where things can get a bit tricky. HTTP Status Codes HTTP status codes are like a first aid kit: they're handy, but they won't fix everything. They give you a broad idea of what's gone wrong, which can help plenty of tools and developers make reasonable assumptions, like: 400 Bad Request: Report error to developers, something is broken. 401 Unauthorized: Might need to refresh a token, don't try again until you have. 404 Not Found: If accepting user input to lookup a resource then this isn't a problem, so don't worry about it. Just tell the user the thing they're looking for isn't there. 405 Method Not Allowed: Ahhhh panic, the API has changed or the client was built wrong. 429 Too Many Requests: Do not retry this request until after the rate limit is over or you'll DDoS the server and get banned. 501 Not Implemented: Oh heck you've gone live relying on an endpoint which isn't ready in production, alert everyone. 504 Gateway Timeout: Probably retry that one straight away as it's likely a network blip. HTTP status codes can convey a lot of assumptions, but they cannot possibly cover all situations, so it's important to add something for the human developers to see what's wrong. Written Description of the Problem Let's say you're building a carpooling app and you need to plan a trip between two places to find more riders. If the coordinates you provide are too close together, the API might respond with something like: HTTP/1.1 400 Bad Request { "error": "Too close for a carpool to be organized, suggest get out and walk." } This is a 400 Bad Request, but that's a pretty common error and a little more information needs to be conveyed, so a string has been added explaining the problem. Next a user tries to plan ## 22 years later, YAML now has a media type DevFeed: [22 years later, YAML now has a media type]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2024-02-20T15:00:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [YAML](), [HTTP](), [API](), [interoperability](), [OpenAPI Specification](), [Internet Engineering Task Force (IETF)](), [Security](), [Rails]() Tags: [api](), [http](), [ietf](), [interoperability](), [openapi](), [security](), [specifications](), [standard](), [standards](), [yaml]() ### AI overview The article explains that RFC 9512 formally registers application/yaml as the media type for YAML and defines +yaml as a structured suffix for YAML-based media types. It discusses the implications for HTTP metadata, OpenAPI specifications, interoperability, existing application conventions, and security considerations. ### Source excerpt As of February 14th 2024, RFC 9512 formally registers application/yaml as the media type for all YAML content, and adds +yaml as a standard structured suffix for all YAML-based more specific media types. With this registration, it's now included in the official media types list maintained by the IANA. Media types like this (also known as the MIME types, from their original invention for email attachment metadata) are heavily used particularly in HTTP Content-Type headers for both requests & responses, and in all sorts of file metadata and processing logic elsewhere. These names give applications a common vocabulary to describe data when passing it around. The additional +yaml suffix also defined here is particularly useful. Media type structured suffixes like this (+xml and +json are other common examples) are used to define specific types for content that's based on an existing generic content type (such as YAML). In this case, this notably opens the door to standardization of other YAML-based MIME types, such as application/openapi+yaml (for OpenAPI specifications that are written in YAML) which is currently being formalized in another standard, following closely behind this one. While a few applications have been using application/yaml and +yaml like this already, many haven't (e.g. Rails still uses application/x-yaml, and others like text/yaml and even text/x-yaml are frequently seen in the wild) and there's never been clear agreement on exactly how this should work. Hopefully with this RFC we'll be able to start picking a single media type consistently from now on, though updating older applications here will obviously take some time. The full RFC is well worth a read if you're interested in the finer details, and discusses all sorts of more detailed questions around interoperability for an evolving language like YAML, its relationship with JSON, and the (many) security considerations to be aware of when defining a formal API around YAML data. This RFC is just ## What is X-Forwarded-For and when can you trust it? DevFeed: [What is X-Forwarded-For and when can you trust it?]() Original publisher: [Read original article]() Author: HTTP Toolkit; Phil Sturgeon Published: 2024-01-31T17:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [HTTP](), [client](), [servers](), [backends](), [Load Balancing](), [Authentication]() Tags: [apis](), [authentication](), [backend](), [http](), [ipv4](), [ipv6](), [load-balancer](), [proxy](), [request](), [server](), [servers](), [standards]() ### AI overview This article explains how the X-Forwarded-For HTTP header records the source IP addresses of clients across proxies, CDNs, load balancers, and other intermediaries. It also describes common uses such as authentication checks, traffic distribution, data localization, and geographic content delivery. ### Source excerpt The X-Forwarded-For (XFF) HTTP header provides crucial insight into the origin of web requests. The header works as a mechanism for conveying the original source IP addresses of clients, and not just across one hop, but through chains of multiple intermediaries. This list of IPv4 and IPv6 addresses is helpful to understand where requests have really come from in scenarios where they traverse several servers, proxies, or load balancers. A typical HTTP request goes on a bit of a journey, traversing multiple layers of infrastructure before reaching its destination. Without the X-Forwarded-For header, the receiving server would only see the IP address of the last intermediary in the chain (the direct source of the request) rather than the true client origin. {/* sequenceDiagram actor Client as Client 28.178.124.142 participant CDN as CDN 198.40.10.101 participant LB as Load Balancer 198.40.10.102 participant B as Backend App 10.0.0.1 Client->>CDN: CDN->>LB: LB->>B: */} In thie example, by the time the backend application is seeing an incoming request, the IP address of the original client is long forgotten. This is where the X-Forwarded-For header can help out. It looks like this: X-Forwarded-For: 28.178.124.142, 198.40.10.101 The goal here is to give a proxy the chance to say "Alright hang on, I'm forwarding you a request, and this is the history of where it came from, as far as I know". Note that the last proxy will not add its own IP address to the list, because that's already available: if the receiver of the request cares about who is calling it directly, they can combine the X-Forwarded-For with the request's source IP address from the incoming connection, e.g: req.connection.remoteAddress in NodeJS. In this example above, the load balancer has said "Hey backend app, I am forwarding you a request that came from this client, via the CDN", and it doesn't need to pop its own IP in there because the backend app can already tell if it's coming from the load balancer or ## Working with the new Idempotency Keys RFC DevFeed: [Working with the new Idempotency Keys RFC]() Original publisher: [Read original article]() Author: HTTP Toolkit; Phil Sturgeon Published: 2023-12-12T15:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [HTTP](), [Internet Engineering Task Force (IETF)](), [client](), [Server](), [stripe]() Tags: [apis](), [http](), [ietf](), [payment](), [request](), [server](), [standards](), [stripe](), [timeout]() ### AI overview This article explains how idempotency keys make retries safe for HTTP API operations, especially POST and PATCH requests that modify server state. It discusses an IETF draft RFC and the risks of duplicate actions such as payments when clients time out. ### Source excerpt Idempotency is when doing an operation multiple times is guaranteed to have the same effect as doing it just once. When working with APIs this is exceptionally helpful on slow or unreliable internet connections, or when dealing with particularly sensitive actions such as payments, because it makes retrying operations safe and reliable. This is why most payment gateways like Stripe and Adyen support 'idempotency keys' as a key feature of their APIs. Recently, the IETF have gone further, and created a draft RFC standard for this useful common pattern, as part of the 'Building Blocks for HTTP APIs' working group. This is technically still a draft and the details could change, but it's fairly mature now and increasingly widely used, so it's a good time to take a closer look, and start using & implementing it for yourself. Idempotency in HTTP APIs Many HTTP methods are defined as idempotent in all cases. In theory, any GET, HEAD, PUT, DELETE, OPTIONS, or TRACE operation can be executed multiple times without any unintended side effects (though for badly behaved APIs your mileage may vary). The idea is that an HTTP request like DELETE /users/123 clearly wants to delete that user, and if that accidentally happens twice then that's just fine. User 123 ends up deleted just the same. It's a lot more complicated for POST and PATCH requests, which do not provide that same level of confidence out of the box. These are designed to allow non-idempotent operations, like adding a new user, sending a payment, or appending to existing data. Those are important use cases too - sometimes you really do want to send the same thing twice, and have it happen twice - but this can cause problems when things go wrong. When sending POST and PATCH requests to modify server state, if you want to support retries then both the client and server need to explicitly handle this - which is exactly what idempotency keys are designed to allow you to do. How can non-idempotency go wrong? Without idempoten ## A brief introduction to OpenAPI DevFeed: [A brief introduction to OpenAPI]() Original publisher: [Read original article]() Author: HTTP Toolkit; Phil Sturgeon Published: 2023-11-28T09:30:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [OpenAPI Specification](), [API](), [HTTP](), [JSON Schema](), [Documentation](), [JSON](), [SDKs](), [Testing]() Tags: [api](), [api-documentation](), [apis](), [graphql](), [http](), [json-schema](), [openapi](), [sdks](), [standards](), [swagger](), [validation]() ### AI overview An introduction to OpenAPI as a machine-readable description format for HTTP APIs. The article explains how OpenAPI describes endpoints, requests, responses, authentication, and validation, and how it supports documentation, testing, validation, SDK generation, server stubs, and mock servers. ### Source excerpt It's hard to work on APIs without hearing about OpenAPI. OpenAPI is an API description format, which is essentially metadata that describes an HTTP API: where it lives, how it works, what data is available, and how it's authenticated. Additional keywords can be used to provide all sorts of validation information, adding a type system to what would otherwise just be arbitrary JSON flying around the internet. OpenAPI has been around for donkeys years, previously known as Swagger but renamed to OpenAPI in 2016. It's powered by JSON Schema, which is also pretty popular in certain circles, but it's only in the last few years that OpenAPI has solidified its place as the description format for HTTP APIs, pushing aside others like RAML and API Blueprint. Elder developers will remember working with WSDLs and XML Schema, and gRPC and GraphQL folks might be thinking "hang on this sounds a bit familiar", and absolutely. Type systems for APIs are pretty common, but here's an excellent one you can use for your REST/RESTish API. Here's an example to give you an idea: openapi: 3.1.3 info: title: Your Awesome API version: '1.0.3' description: More information and introduction. paths: /things: post: summary: Create a thing requestBody: content: application/json: schema: type: object properties: name: type: string examples: - Tim responses: '201': description: "Created" content: application/json: schema: type: object properties: id: type: string format: uuid name: type: string examples: - Tim created_at: type: string format: date-time example: 2020-01-01T00:00:00Z This describes an API in a machine-readable format, including overall metadata, endpoint paths, request formats, and the details of possible responses you might receive. What can OpenAPI do? OpenAPI specifications provide a machine-readable base on top of which lots of neat API tools can be used and even generated. One of the most common uses by many API teams to to generate API reference documentation, which helps end-users ## 6 Ways to Debug a Failing Docker Container DevFeed: [6 Ways to Debug a Failing Docker Container]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-10-31T14:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [Docker](), [debugging](), [container](), [Command-line interface]() Tags: [command-line](), [container](), [debugging](), [docker](), [errors](), [logs](), [metrics]() ### AI overview A tutorial on diagnosing Docker containers that fail to start or behave unexpectedly. It covers commands for viewing logs and resource usage, copying files, opening a shell, and inspecting a stopped container by creating an image with a different command. ### Source excerpt Everything crashes. Sometimes things crash when they're running inside a Docker container though, and then all of a sudden it can get much more difficult to work out why, or what the hell to do next. Docker's great, but it's an extra layer of complexity that means you can't always easily poke your app up close any more, and that can really hinder debugging when your container fails to start or breaks in unusual ways. If you're stuck in that situation, here are my goto debugging commands to help you get a bit more information on exactly what's up: docker logs Hopefully you've already tried this, but if not: start here. This'll give you the full STDOUT and STDERR command-line output from the command that was run initially in your container. You can also use docker attach to stream the live logs from an active container, if you want to keep an eye on the output as it runs. docker stats If you just need to keep an eye on the metrics of your container to work out what's gone wrong, docker stats can help: it'll give you a live stream of resource usage, so you can see just how much memory you've leaked so far and easily spot if your CPU usage is way out of control. docker cp :/path/to/useful/file /local-path Often just getting hold of more log files is enough to sort you out. If you already know what you want, docker cp has your back: copy any file from any container back out onto your local machine, so you can examine it in depth (especially useful analysing heap dumps). docker exec -it /bin/bash Next up, if you can run the container (if it's crashed, you can restart it with docker start ) then you can use this command to oppen a command line shell inside the container directly, and start digging around for further details by hand. docker commit my-broken-container && docker run -it my-broken-container /bin/bash Can't start your container at all? If your container starts ## New ways to inject system CA certificates in Android 14 DevFeed: [New ways to inject system CA certificates in Android 14]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-09-21T12:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [Android](), [HTTP](), [Emulator]() Tags: [adb](), [android](), [article](), [certificates](), [interception](), [tls]() ### AI overview This article explains that Android 14 blocks the previous direct method of installing system-level CA certificates, including on rooted devices, but describes newer practical approaches for rooted Android 14+ devices. It also distinguishes system-level certificate injection from enterprise-managed and user-installed certificates, and mentions HTTP Toolkit's ADB-based automatic setup. ### Source excerpt A couple of weeks ago I published a post about changes in Android 14 that fundamentally break existing approaches to installing system-level CA certificates, even with root access. This has triggered some fascinating discussion! I highly recommend a skim through the debate on Mastodon and Hacker News. Since that was posted, quite a few people have talked to me about possible solutions, going beyond the previous approaches with new mechanisms that make it practical to do this in Android 14+, and there are some good options here. While direct root access to change these certificates by simply writing to a directory is indeed no longer possible, root is root, and so with a bit of work there are still some practical & effective ways to dig down into the internals of Android and seize control of these certs once more. Choose your own adventure: If you just want to intercept an Android 14+ device right now, stop reading this, download the latest HTTP Toolkit, connect your device to ADB, click the 'Android Device via ADB' interception option for automatic setup, and dive into your traffic. If you just want to know the steps to manually do system certificate injection on Android 14 for yourself, jump down to How to install system CA certificates in Android 14. If you want the full background, so you can understand how & why this all works, read on: Clearing up confusion Before digging into this, I do want to explicitly clear up a few misunderstandings that I've seen repeatedly pop up from the previous article: These changes don't affect installation of CAs in other scenarios. As far as I'm aware, CA installation for fully managed enterprise-provisioned devices and the limited user-installed (as opposed to system-level) CA certificates will continue functioning as before. If you're not using root access to inject system-level CA certificates into a rooted device or emulator, you don't need to worry about this. Similarly, it is still possible to soft-remove system CA certific ## Android 14 blocks modification of system certificates, even as root DevFeed: [Android 14 blocks modification of system certificates, even as root]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-09-05T14:00:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [Android](), [Security](), [TLS (Transport Layer Security)](), [API]() Tags: [android](), [android-14](), [api](), [certificates](), [interception](), [root](), [security](), [tls]() ### AI overview The article examines Android 14's tighter restrictions on modifying trusted certificate authorities, including on rooted devices, and the resulting challenges for developers, testers, and researchers. An update notes that multiple workarounds now allow certificate injection. ### Source excerpt Update: This post sparked a lot of excellent discussion and debate on workarounds, and there are now multiple working solutions to allow certificate injection on Android 14, despite the restrictions discussed here. See the update post for more details. When Android was initially announced in 2007 by the Open Handset Alliance (headed by Google) their flagship project was billed as an "open platform", "providing developers a new level of openness", and giving them "complete access to handset capabilities and tools". We've come a long way since then, steadily retreating from openness & user control of devices, and shifting towards a far more locked-down vendor-controlled world. The next step of Android's evolution is Android 14 (API v34, codename Upside-Down Cake) and it takes more steps down that path. In this new release, the restrictions around certificate authority (CA) certificates become significantly tighter, and appear to make it impossible to modify the set of trusted certificates at all, even on fully rooted devices. If you're an Android developer, tester, reverse engineer, or anybody else interested in directly controlling who your device trusts, this is going to create some new challenges. Before we get into the finer details, first I want to talk a little about the context around Android CA management and how we got here, but if you want to jump to the latest details you can go straight to the Enter Android 14 section below. "Open Software, Open Device, Open Ecosystem" While the initial principles of Android were very much focused on open software, controllable by users and developers, over more recent years Android has increasingly limited the control of users, developers & researchers over their own devices. The key turning point in this process was Android 7 (Nougat, released in 2016) in which the certificate authorities (CAs) on the device that were previously fully modifiable by the owner of the phone were split in two: one fixed list of CAs provided ## Apple already shipped attestation on the web, and we barely noticed DevFeed: [Apple already shipped attestation on the web, and we barely noticed]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-07-25T14:00:00Z Content type: opinion Language: en Sources: [HTTP Toolkit]() Topics: [Web](), [browser](), [HTTP](), [iOS](), [macOS](), [Cloudflare]() Tags: [apple](), [attestation](), [browser](), [browsers](), [chromium](), [cloudflare](), [http](), [ios](), [macos](), [public-key](), [server](), [web]() ### AI overview This opinion article examines Apple's Private Access Tokens, an attestation system integrated into macOS 13, iOS 16, and Safari. It explains how browsers, operating systems, attesters, and token issuers use HTTP challenges and signed tokens to verify that requests come from legitimate devices without disclosing the user's identity, while comparing the system with the proposed Web Environment Integrity model. ### Source excerpt There's been a lot of concern recently about the Web Environment Integrity proposal, developed by a selection of authors from Google, and apparently being prototyped in Chromium. There's good reason for anger here (though I'm not sure yelling at people on GitHub is necessarily the best outlet). This proposal amounts to attestation on the web, limiting access to features or entire sites based on whether the client is approved by a trusted issuer. In practice, that will mean Apple, Microsoft & Google. Of course, Google isn't the first to think of this, but in fact they're not even the first to ship it. Apple already developed & deployed an extremely similar system last year, now integrated into MacOS 13, iOS 16 & Safari, called "Private Access Tokens": Private Access Tokens are powerful tools that prove when HTTP requests are coming from legitimate devices without disclosing someone's identity. The focus here is primarily on removing captchas, and as such it's been integrated into Cloudflare (discussed here) and Fastly (here) as a mechanism for recognizing 'real' clients without needing other captcha mechanisms. Fundamentally though, it's exactly the same concept: a way that web servers can demand your device prove it is a sufficiently 'legitimate' device before browsing the web. How do Private Access Tokens work? The mechanism is a fairly simple exchange over HTTP, handled by built-in browser APIs, which in turn integrate with operating system components to confirm that the browser & OS are 'legitimate' (the exact definition of that is left to the attester - i.e. Apple). The flow looks like this: A browser makes an HTTP request from a web server. The web server refuses the request, and returns an HTTP 401 response with a PrivateToken challenge: HTTP/1.1 401 Unauthorized WWW-Authenticate: PrivateToken challenge=, token-key= (Newlines added for readability) The browser recognizes this, and sends parts of the challenge, in addit ## Bunny.net CDN caching flaw exposed authenticated users' private HTTP responses DevFeed: [Bunny.net CDN caching flaw exposed authenticated users' private HTTP responses]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-06-20T11:30:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [Caching](), [HTTP](), [Cross-origin resource sharing (CORS)](), [Authentication](), [web applications]() Tags: [authentication](), [caching](), [cdn](), [http](), [performance](), [protection](), [responses](), [security]() ### AI overview The article examines a Bunny.net CDN caching flaw that caused private HTTP responses intended for one authenticated user to be served to other users. It explains how CDN caching can diverge from HTTP standards and expose private data or authentication credentials, and notes that the issue has been fixed. ### Source excerpt Caching is hard. Unfortunately though, caching is quite important. Hosted caching & CDNs offer incredible powers that can provide amazing performance boosts, cost savings & downtime protection, essential for most modern sites with any serious volume of users. Unfortunately, while there are strict standards for how caching is supposed to work with HTTP on the web, many cache providers do not quite follow these, instead giving their customers free reign over all kinds of invalid caching behaviour, and providing their own default configurations that often don't closely follow these standards to start with either. There are many good reasons for this, but the main one is that CDNs are now doing dual service: providing performance improvements, and actively protecting upstream sites from DoS attacks and traffic spikes (similar problems - the key difference between a DoS attack and hitting #1 on Hacker News etc is intent, not impact). This conflicts with many of the standards, which prioritize correctness and predictability over this use case and, for example, expect clients to be able to unilaterally request that the cache be ignored. Bunny.net provides one of these CDNs, and like most they aggressively cache content beyond the limits of the standards, both to help protect upstream servers and to support advanced user use cases. This has upsides, but in some edge cases can result in awkward bugs and break developer expectations. In some more dramatic cases though, it can expose private user data, break applications & even leak auth credentials, and that's where this story gets serious. A few months ago, I ran into exactly that issue while testing out deployment options with Bunny.net, where I discovered that private HTTP responses intended for one authenticated user could be served to other users instead. Spoiler: this is now fixed! That said, it's worth exploring where this went wrong, the many ways this can work right, and how CDNs solve issues like this in practice. C ## Automatic npm publishing, with GitHub Actions & npm granular tokens DevFeed: [Automatic npm publishing, with GitHub Actions & npm granular tokens]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-03-22T10:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [npm](), [GitHub Actions](), [GitHub](), [Security](), [ci]() Tags: [ci](), [github](), [github-actions](), [javascript](), [node-js](), [npm](), [releases](), [secrets](), [security]() ### AI overview This tutorial explains how to automate npm package publishing with GitHub Actions using npm granular access tokens. It describes restricting tokens to specific packages, storing them as GitHub Actions secrets, allowing publishing without 2FA for granular or automation tokens, and configuring a publishing workflow. ### Source excerpt This week, at long last, GitHub announced granular access tokens for npm. This is a big deal! It's great for security generally, but also particularly useful if you maintain any npm packages, as it removes the main downside of automating npm publishing, by allowing you to give CI jobs only a very limited token instead of full 2FA-free access to your account. In the past, I've wished for this, because I maintain a fair few npm packages including some very widely used ones. The previous solution of "just disable 2FA on your account, create an all-powerful access token with global access to every package, and give that token to your CI job" was not a comfortable one. Regardless of your situation, isolating any risk of issues in security-sensitive situations like this is a good move, and ensures that any leak of (or legitimate access to) your CI secrets for one project doesn't imply a complete takeover of everything on your npm account. As soon as I saw this was now available, I jumped on automating npm publishing for a few of the packages that I've been manually publishing until now. The process is pretty quick and easy, let's walk through the steps: Get an access token for your package Log into npmjs.com Click your profile picture in the top right, then 'Access Tokens', 'Generate New Token', and 'Granular Access token' (or jump to npmjs.com/settings/$YOUR_USERNAME/tokens/granular-access-tokens/new) Set a useful name, a long expiry (up to you), 'Read and write' permissions, and pick the specific package that you're publishing Add your token as a secret for your project's GitHub Actions Jump to https://github.com/$YOU/$REPO/settings/secrets/actions/new Set NPM_PUBLISH_TOKEN as the secret name Copy the npm_... token from the previous step as the secret value In your npm package's settings (i.e. https://www.npmjs.com/package/$PACKAGE_NAME/access), allow publish without 2FA for granular/automation tokens only, so that tokens can be used for publishing: Add a publish step t ## Docker Hub policy changes create risks for community image publishers DevFeed: [Docker Hub policy changes create risks for community image publishers]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-03-17T11:00:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [Docker Hub](), [Docker](), [Self-hosted](), [hosting](), [Monitoring]() Tags: [docker](), [docker-registry](), [monitoring](), [registry](), [self-hosting]() ### AI overview This commentary examines Docker Hub policy changes affecting community-run image publishers. It explains the impact on publishing control and considers self-hosting or migrating to another hosted Docker registry. ### Source excerpt As you may have seen, Docker Hub made a dramatic shift in policy this week, and effectively gave a 30 day eviction notice to almost all community-run images. They've now made an apology to 'clarify' a few details, and helpfully take some of the hard edges off, but this still highlights a big problem. Fortunately, there are solutions. As initially described, this would've been catastrophic: Docker Hub has been used as the default host in tooling, tutorials, demos, blog posts, scripts, deployment definitions, CI builds and more for many years, and all those references were going to break - a self-inflicted left-pad for the Docker ecosystem. In their updated policy, it appears they now won't remove any existing images, but projects who don't pay up will not be able to publish any new images, so they've effectively lost control of the namespace they used to deploy to their communities unless they purchase a full team subscription. Many will not do so. This is an interesting challenge. Even if the existing images aren't removed, the direction of travel for Docker Hub is now clear: they don't want to host the core of the Docker community any more, no more freebies, pay up or go elsewhere (not unreasonable, but something of a rugpull after a full decade of the opposite). As a small/open-source/community/hobbyist image publisher, or if you depend on Docker Hub for free image publishing in any capacity, you now have a problem. They don't want you there. You're explicitly not their audience, and the rules will likely tighten further. This isn't unreasonable - it's their service and hosting isn't free - but it's worth considering explicitly and reacting accordingly. If you're not a paying Docker Hub customer, it's time to leave Docker Hub. The hard part is what to do instead. Self-hosting a registry is not free, and it's more work than it sounds: it's a proper piece of infrastructure, and comes with all the obligations that implies, from monitoring to promptly applying securit ## EU Funding for Mobile App Traffic Interception DevFeed: [EU Funding for Mobile App Traffic Interception]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2023-02-27T12:00:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [interception](), [Mobile](), [Security & Privacy](), [Android](), [iOS](), [Instrumentation](), [Open Source](), [Network]() Tags: [android](), [announcements](), [certificates](), [development](), [eu](), [frida](), [funding](), [hooks](), [http](), [instrumentation](), [interception](), [ios](), [mobile](), [mobile-apps](), [network](), [open](), [privacy](), [product-development](), [security](), [technical]() ### AI overview HTTP Toolkit announces EU-backed funding to improve HTTPS traffic interception for mobile apps on iOS and Android. The project will use Frida integrations and hooks to support app-specific interception, automatic certificate unpinning, and simpler setup. ### Source excerpt HTTP Toolkit has been selected to receive another round of open-source funding from the EU! This aims to improve interception of HTTPS traffic from mobile apps, making it easier for both security/privacy researchers and normal technical users to inspect & manipulate the data that any app they use sends & receives. This funding will directly support work to improve the precision & usability of mobile app interception, on both iOS and Android. In a couple of clicks, you'll be able to target any app installed on a connected device (on either platform) and inspect all its traffic. That means no background noise from full system interception, automatic certificate unpinning, and no fiddly manual setup required. HTTP Toolkit can already automatically intercept Android devices, but only globally for the whole device (using Android's VPN APIs), with system certificates injected but without certificate unpinning, and with no automatic setup support for iOS at all. All those caveats are going away (of course, I'm intending the existing device-wide interception to remain a fully supported option indefinitely too). This is going be powered by a set of new integrations & hooks for Frida, a popular open-source instrumentation toolkit. Many advanced HTTP Toolkit users are already using Frida independently (see the Frida certificate unpinning guide for more info) but doing so often requires quite a bit of setup and specialist knowledge. This project is going to take that away, making network interception of mobile apps easily accessible to anybody who knows what "HTTP" is. All this is being funded by the fantastic NLNet Foundation as part of NGI Zero Entrust, a program funding open-source EU projects that support transparency around data use & privacy in modern technology. This is part of the EU's Next Generation Internet (NGI) initiative, aiming to directly fund researchers & open-source developers to encourage the future engineering of the Internet towards European values: "openn ## Debugging WebRTC, IPFS & Ethereum with HTTP Toolkit DevFeed: [Debugging WebRTC, IPFS & Ethereum with HTTP Toolkit]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2022-10-28T13:30:00Z Content type: article Language: en Sources: [HTTP Toolkit]() Topics: [WebRTC](), [Ethereum](), [IPFS](), [debug](), [web applications](), [browsers](), [HTTP]() Tags: [blockchain](), [browsers](), [debug](), [debugging](), [decentralized-web](), [ethereum](), [http](), [interception](), [javascript](), [web](), [webrtc]() ### AI overview This article describes HTTP Toolkit support for capturing, debugging, mocking, rewriting, and injecting errors in WebRTC, IPFS, and Ethereum traffic used by decentralized web applications. It also explains why debugging multi-protocol applications matters and discusses related protocols used alongside HTTP. ### Source excerpt HTTP is important on the web, but as other alternative protocols grow popular in networked applications, it's often important to be able to capture, debug and mock those too. I've been working on expanding HTTP Toolkit's support for this over the past year (as one part of a project funded by EU Horizon's Next Generation Internet initiative), to extend HTTP Toolkit to cover three additional rising protocols that are often used alongside simple HTTP in decentralized web applications: WebRTC, IPFS & Ethereum. This is now live! If you're using HTTP Toolkit to intercept browsers, and a web application connects to another peer over WebRTC, interacts with the Ethereum blockchain, or pulls content from the IPFS network, then you'll now see this inline amongst your collected HTTP traffic, and you can create rules to rewrite these requests, define mock responses, or inject errors. Let's talk about why this matters, what you can do with this, and how it actually works internally: Why? HTTP remains the key protocol on the web, by a wide margin, and that's not going to change any time soon. That said, there are an increasing number of new protocols being explored both alongside and on top of HTTP, covering alternate use cases and supporting different communication models, and as software evolves this will only increase. HTTP Toolkit needs to be able to support these, to ensure that you can easily understand and test all communications from the increasing number of multi-protocol applications. There's a lot of examples of HTTP-adjacent protocols where this applies: WebSockets, which grew to cover use cases needing persistent connections that weren't well supported by the request/response model, which use HTTP just for initial connection setup, and are now widely supported and used in HTTP-based apps. WebTransport, a new QUIC-based connection protocol that's aiming to supplant WebSockets with a modern approach that adds a swathe of other benefits on top. GraphQL, a general-purpose ## Testing libraries for the Decentralized Web DevFeed: [Testing libraries for the Decentralized Web]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2022-10-24T17:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [Testing](), [IPFS](), [Ethereum](), [Integration testing](), [ci](), [WebRTC](), [JavaScript](), [P2P](), [web applications]() Tags: [ci](), [data](), [database](), [decentralized-web](), [ethereum](), [hosting](), [integration-testing](), [javascript](), [testing](), [web-applications](), [webrtc]() ### AI overview This article introduces MockIPFS and Mockthereum, network interception libraries for IPFS and Ethereum. It explains how they support automated integration testing and CI workflows while also serving as foundations for network proxy tools for decentralized web applications. ### Source excerpt The world of decentralized web applications is an exciting place that has exploded in recent years, with technologies such as IPFS and Ethereum opening up possibilities for a peer-to-peer web - creating applications that live outside the traditional client/server model, where users to interact and control their own data directly. At the same time, it's still immature, and for software developers it lacks a lot of the affordances & ecosystem of the traditional HTTP-based web app world. There's far fewer tools and libraries for developers working in this space. I've been working on improving this over the last year (as one part of a project funded by EU Horizon's Next Generation Internet initiative), by building network interception libraries for both IPFS & Ethereum: MockIPFS & Mockthereum. These each act as both an immediately useful automated testing library, to support modern integration testing & CI workflows, and a base for building more general network proxy tools for web applications using either (or both) technologies. If that sounds cool and you just want to jump straight in and try these for yourself, you can get started at github.com/httptoolkit/mockipfs/ and github.com/httptoolkit/mockthereum/. On the other hand, if you want to hear what this can do in practice, and learn a little about how it works under the hood, read on: A new way to build web apps Decentralized web apps often use a mix of many different technologies, at various layers of the stack, such as: IPFS - for decentralized static content hosting & data storage Ethereum - for decentralized consistent global state, computation on that state, and financial transactions Filecoin/Storj - for paid decentralized long-term content storage WebRTC - for peer-to-peer raw data transfer, and video/audio connections Service workers - a JavaScript API allowing fully offline web apps Handshake (HNS)/Ethereum Name System (ENS) - to map domain names to web applications GunDB - a decentralized database for the ## How to intercept, observe & mock WebRTC traffic DevFeed: [How to intercept, observe & mock WebRTC traffic]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2022-10-13T10:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [WebRTC](), [Tooling](), [P2P](), [Security & Privacy]() Tags: [debugging](), [developer-tools](), [how-to](), [interception](), [javascript](), [p2p](), [security-privacy](), [webrtc]() ### AI overview This tutorial explains why WebRTC traffic is difficult to inspect and mock, covering its peer-to-peer encryption, negotiated networking, and limited debugging APIs. It introduces MockRTC and describes how controlling one peer can enable interception, observation, and transformation of WebRTC traffic. ### Source excerpt WebRTC allows two users on the web to communicate directly, sending real-time streams of video, audio & data peer-to-peer, from within a browser environment. It's exciting tech that's rapidly maturing, already forming the backbone of a huge range of video chat, screen sharing and live collaboration tools, but also as a key technology for decentralization of web apps - providing a P2P data transport layer used by everything from WebTorrent to IPFS to Yjs. Unfortunately though, it doesn't have the tooling ecosystem that developers used to networking with HTTP often expect. There's few supporting tools or libraries, inspecting raw traffic is hard or impossible, and mocking WebRTC traffic for automated testing is even harder. Even built-in low-level browser tools like chrome://webrtc-internals don't allow seeing messages sent on WebRTC data channels. It's hard to build modern secure web applications on top of protocols that you can't directly see or interact with. This doesn't just affect developers: it also seriously impacts security & privacy researchers and reverse engineers, each trying to investigate the traffic sent & received by the apps we all use. If you want to know what data a webapp you use is sending over WebRTC, right now it's very hard to find out. Intercepting WebRTC traffic to build these tools and libraries is difficult, because unlike protocols like HTTP that were designed to allow active proxying and user-configureable PKI (i.e. CA certificates) early on, WebRTC encrypts all traffic using peer-to-peer negotiated certificates for authentication without PKI, communicates in a wide variety of different negotiated ways at the network level to avoid NAT issues, and offers no convenient APIs to configure this for debugging. All of this provides some great features to the protocol as a user, but some serious challenges when building developer tools. As it turns out though, despite this, there are just enough places where we can hook into that it is possible ## Chrome 99 expands Certificate Transparency on Android, affecting HTTPS interception and debugging tools DevFeed: [Chrome 99 expands Certificate Transparency on Android, affecting HTTPS interception and debugging tools]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2022-05-11T16:00:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [Certificate Transparency](), [Chrome](), [Android](), [Security](), [Tooling]() Tags: [android](), [browsers](), [certificate-transparency](), [chrome](), [developer-tools](), [interception](), [security](), [tls]() ### AI overview This article explains how Chrome 99's expansion of Certificate Transparency to all Android Chrome users affects HTTPS interception with trusted system CA certificates, including developer debugging tools. It describes why Certificate Transparency exists and discusses workarounds for debugging Chrome traffic on Android. ### Source excerpt Certificate transparency is superb improvement to HTTPS certificate security on the web that's great for users and businesses, but on Android it creates a huge problem for the many developer tools like HTTP Toolkit which install trusted system certificates into Android to intercept & debug app traffic. This doesn't appear in the main announcements anywhere, but buried deep in the enterprise release notes for Chrome v99 there's a small note that says: Certificate transparency is already enforced on desktop platforms, and for some Android users. Chrome 99 expands certificate transparency to all Android Chrome users. And with that small note, Chrome on Android become uninterceptable for all HTTP Toolkit users using rooted devices, and anybody else who actively installs and trusts their own system CA certificates. If you're running into an ERR_CERTIFICATE_TRANSPARENCY_REQUIRED error in Chrome while trying to debug your HTTPS traffic with some MitM debugging proxy, then this is affecting you too. Let's talk about how certificate transparency works, why this breaks, and how you can work around it to keep debugging HTTPS from Chrome on your Android device regardless. Certificate Transparency (CT) HTTPS certificates are issued and signed by Certificate Authorities (CAs) who are trusted by your browser & OS. That's great when it works, but sometimes it doesn't. CAs can make mistakes when issuing certificates, when verifying a client's identity beforehand, or through malice somewhere, and issue fraudulent certificates to people who shouldn't have them. For example, let's say a trusted CA issues a certificate for google.com to the wrong person (this actually happened, repeatedly). That issued certificate is incredibly powerful - whoever has it can freely intercept all traffic sent by anybody to Google.com and both see & modify that traffic, whilst browsers will show all users a padlock and tell them everything is totally fine & secure. Even worse though: attacks like this were ## Fighting TLS fingerprinting with Node.js DevFeed: [Fighting TLS fingerprinting with Node.js]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2021-12-07T13:40:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [TLS (Transport Layer Security)](), [Node.js](), [Web Scraping](), [HTTP]() Tags: [browser](), [http](), [interception](), [javascript](), [mitm](), [node-js](), [proxy](), [tls]() ### AI overview This tutorial explains how TLS fingerprinting lets servers and proxies identify client software from unencrypted connection metadata. It examines how this can block HTTP Toolkit and other automated traffic, then shows how Node.js techniques can work around that blocking. ### Source excerpt The modern internet is full of services that want to know who you are. Fingerprinting is the latest way to do this: capturing many small details about your client, and using it to create an id that's sufficiently unique to recognize you and infer details about your network client and device. This is a privacy problem, which I'm not going to focus on here, but collecting and analysing interaction metadata is also a powerful tool to recognize certain types of clients, even when they attempt to disguise themselves. TLS provides a particularly good surface for this kind of fingerprinting, which allows a server or proxy to recognize the kind of software (a specific browser version, Python, Ruby, Node.js, etc) that's opening any TLS connection, before the client has even sent any data (such as an HTTP request) within the connection, and purely using unencrypted public data from the connection. In many cases, this is a problem. HTTP Toolkit acts as a MitM proxy for HTTP(S) traffic inspection & mocking, and this potentially allows servers to recognize and block it, along with any other similar debugging proxies. Many other automated scripts and tools can also be recognized, blocking web scraping and other requests from anything but a real browser. Until recently, I thought this was fairly theoretical, but last week an HTTP Toolkit user showed me a real-world example, where non-browser traffic is blocked completely, based just on its TLS fingerprint, causing big problems for HTTP Toolkit usage. Fortunately, we can work around this. In this article, I want to explain how TLS fingerprinting works, look at a real-world example, and then see exactly how you can defeat this blocking using Node.js (with techniques that you can easily apply elsewhere too). How does TLS fingerprinting work? TLS provides a huge amount of data for fingerprinting. Every connection secured by TLS (for example, all HTTPS requests) starts with a 'client hello' message from the client, sent unencrypted, wi ## Reverse engineering & modifying Android apps with JADX & Frida DevFeed: [Reverse engineering & modifying Android apps with JADX & Frida]() Original publisher: [Read original article]() Author: HTTP Toolkit; Tim Perry Published: 2021-11-22T12:30:00Z Content type: tutorial Language: en Sources: [HTTP Toolkit]() Topics: [Reverse Engineering](), [Android](), [Security](), [App]() Tags: [android](), [certificates](), [frida](), [http](), [interception](), [jadx](), [reverse-engineering](), [security]() ### AI overview A practical guide to reverse engineering and modifying Android applications using JADX and Frida. It introduces the initial workflow for locating and understanding relevant code, using certificate pinning as an example for disabling runtime checks so HTTP Toolkit can intercept HTTPS traffic. ### Source excerpt I get a lot of emails from users who want to know exactly what their favourite Android app is doing, and want to tweak and change how that works for themselves. There are some great tools to do this, including JADX & Frida, but using these is complicated, and every reverse engineering problem has its own unique challenges & solutions. There's few good guides to getting started, and even fewer guides on the advanced tricks available. In this article, I want to talk you through the core initial steps to look inside any Android app, give you the tools to find & understand the specific code that matters to you, and then show you how you can use that information to modify the app for yourself. Let's set the scene first. Context I'm assuming here that somebody else has written an Android app that you're interested in. You want to know exactly how a specific bit of behaviour works, and you want to change what it's doing. I'm going to focus on the classic HTTP Toolkit user example here of certificate pinning: where security-conscious apps that send HTTPS traffic go beyond the normal HTTPS validation requirements, and actively check that the HTTPS certificates used are from a small set of specific trusted certificates, not just the standard set trusted by all Android devices. (I'm focusing on certificate pinning because it's a common use case and it's convenient, but the techniques here work for all other kinds of reverse engineering & patching too, don't worry!) Certificate pinning is a problem for HTTP Toolkit users, who are trying to intercept HTTPS traffic to see what messages their Android apps are sending & receiving. It's not possible to intercept these app's traffic because they won't trust HTTP Toolkit's certificate, even after it's been injected into the device's system certificate store. Using the tools we're going to talk about in a moment we can take an unknown 3rd party app, find the certificate pinning code within it, and disable that remotely while the app ru [Next page]()