Replace guest tokens with account sessions (#1210)

* Replace tokens with guest accounts, swap endpoints

* Filter out account limits that already reset

* Track rate limits, reset after 24 hours

* Add missing limitedAt assignment

* Add guest accounts variable to GitHub action

* Unify some guest account logs

* Fix compilation error

* Reduce max concurrent pending requests per account

* Fix compilation on older versions of Nim

* Tweak /.tokens, add amount of limited accounts

* Switch back to old user search endpoint

* Fix tweetDetail stats

* Switch to using typeahead for user search

* Revert "Switch to using typeahead for user search"

This reverts commit a3e11e3.

* Switch to TweetDetail for tweets

* Cleanup

* Fix cards not being displayed

* Improve .tokens output

* Handle ProtocolError and BadClientError equally

* Improve rate limit handling, minor refactor

* Fix accounts logging

* Add 429 logging

* Add empty oauth token logging

* Fix negative pending requests bug

* Add media proxying error logging

* Shorten media proxy error log

* Retry rate limited requests with different account

* Fix crash on missing quote tweet data crash

* Increase photo rail cache ttl

* Improve guest accounts loading, add JSONL support

* Improve Liberapay css for Firefox compatibility

* fix(nitter): add graphql user search (#1047)

* fix(nitter): add graphql user search

* fix(nitter): rm gitignore 2nd guest_accounts

* fix(nitter): keep query from user search in result. remove personal mods

* fix(nitter): removce useless line gitignore

* Fix multi-user timelines

* Update nimcrypto and jsony

* Bump minimum Nim version

* Replace /.tokens with /.health and /.accounts

* Fix compilation on Nim 1.6.x

* Run tests on multiple Nim versions

* Fix guest accounts CI setup

* Fix guest accounts CI setup attempt 2

* Prevent annoying warnings on devel

* Fix missing CI file argument

* Fix guest accounts CI setup attempt 3

* Remove redundant "active" field from /.health

* Fix guest accounts CI setup attempt 4

* Fix outdated test

* Move CI workflow to buildjet

* Purge expired accounts after parsing

* Don't requests made before reset

* Fix #670

* Bump Nim in the ARM64 Dockerfile, add nitter user

* Fix ARM64 Dockerfile versions

* Fix #1106

Closes #831

* Enable disabled tests

* Add simple job_details card support

* Fix jobDetails error on old Nim versions

* Add support for business and gov verification

Also improve icon rendering on Firefox

* Fix compilation with old Nim again

* Fix search queries in user search RSS feeds (#1126)

Fixes #992

* Fix broken video playback by forcing fmp4

* Fix for #1147, Proxy for audio URL and upgrade hls.js (#1178)

* Revert "Fix broken video playback by forcing fmp4"

This reverts commit 52db03b.

* Fix audio url in video m3u8

* Upgrade hls.js to 1.5.1 and use full version

* Update auth.nim (#1164)

Avoid expiring the tokens for now.

See:
- #983 (comment)
- #1155 (comment)

Thanks @cmj

* Replace old v1 photo rail API with gql

* Raise NoAccountsError when all accounts limited

* Move limited flag to be account-level

* Capture "account locked" API error

* Update authority header

* Add regex for x.com links

* Update endpoint versions, switch tweet endpoint

* Return 403 on hmac error

* Add lazy loading for images

* Fix missing video thumbnail being too small

* Silence 404 proxy errors

* Skip null fetch errors

* Increase max concurrent reqs per account

* Rename accounts/guest accounts to sessions

The new file loaded by default is now ./sessions.jsonl
JSONL is also required, .json support dropped.

* Update Dockerfiles

* Update README, added an important note

* Add Python script to create account sessions

* Change main page search to "Tweets" search

* Remove old tokenCount from nitter.example.conf

* Fix tests

* Fix empty image URLs in photo rail

---------

Co-authored-by: LS <[email protected]>
Co-authored-by: blankie <[email protected]>
Co-authored-by: jackyzy823 <[email protected]>
Co-authored-by: somini <[email protected]>

Author: 5 people

.github/workflows/run-tests.yml

@@ -10,25 +10,34 @@ on:
 
 jobs:
   test:
-    runs-on: ubuntu-latest
+    runs-on: buildjet-2vcpu-ubuntu-2204
+    strategy:
+      matrix:
+        nim:
+          - "1.6.x"
+          - "2.0.x"
+          - "2.2.x"
+          - "devel"
     steps:
       - uses: actions/checkout@v3
         with:
           fetch-depth: 0
       - name: Cache nimble
         id: cache-nimble
-        uses: actions/cache@v3
+        uses: buildjet/cache@v3
         with:
           path: ~/.nimble
-          key: nimble-${{ hashFiles('*.nimble') }}
-          restore-keys: "nimble-"
+          key: ${{ matrix.nim }}-nimble-${{ hashFiles('*.nimble') }}
+          restore-keys: |
+            ${{ matrix.nim }}-nimble-
       - uses: actions/setup-python@v4
         with:
           python-version: "3.10"
           cache: "pip"
       - uses: jiro4989/setup-nim-action@v1
         with:
-          nim-version: "1.x"
+          nim-version: ${{ matrix.nim }}
+          repo-token: ${{ secrets.GITHUB_TOKEN }}
       - run: nimble build -d:release -Y
       - run: pip install seleniumbase
       - run: seleniumbase install chromedriver
@@ -37,9 +46,11 @@ jobs:
         run: |
           sudo apt install libsass-dev -y
           cp nitter.example.conf nitter.conf
+          sed -i 's/enableDebug = false/enableDebug = true/g' nitter.conf
           nimble md
           nimble scss
+          echo '${{ secrets.SESSIONS }}' > ./sessions.jsonl
       - name: Run tests
         run: |
           ./nitter &
-          pytest -n4 tests
+          pytest -n3 tests

.gitignore

@@ -10,4 +10,6 @@ nitter
 /public/css/style.css
 /public/md/*.html
 nitter.conf
+guest_accounts.json*
+sessions.json*
 dump.rdb

Dockerfile

@@ -1,4 +1,4 @@
-FROM nimlang/nim:2.0.0-alpine-regular as nim
+FROM nimlang/nim:2.2.0-alpine-regular as nim
 LABEL maintainer="[email protected]"
 
 RUN apk --no-cache add libsass-dev pcre
@@ -9,7 +9,7 @@ COPY nitter.nimble .
 RUN nimble install -y --depsOnly
 
 COPY . .
-RUN nimble build -d:danger -d:lto -d:strip \
+RUN nimble build -d:danger -d:lto -d:strip --mm:refc \
     && nimble scss \
     && nimble md
 

Dockerfile.arm64

@@ -1,23 +1,25 @@
-FROM alpine:3.18 as nim
+FROM alpine:3.21.2 as nim
 LABEL maintainer="[email protected]"
 
-RUN apk --no-cache add gcc git libc-dev libsass-dev "nim=1.6.14-r0" nimble pcre
+RUN apk --no-cache add libsass-dev pcre gcc git libc-dev nim nimble
 
 WORKDIR /src/nitter
 
 COPY nitter.nimble .
 RUN nimble install -y --depsOnly
 
 COPY . .
-RUN nimble build -d:danger -d:lto -d:strip \
+RUN nimble build -d:danger -d:lto -d:strip --mm:refc \
     && nimble scss \
     && nimble md
 
-FROM alpine:3.18
+FROM alpine:3.21.2
 WORKDIR /src/
-RUN apk --no-cache add ca-certificates pcre openssl1.1-compat
+RUN apk --no-cache add pcre ca-certificates openssl
 COPY --from=nim /src/nitter/nitter ./
 COPY --from=nim /src/nitter/nitter.example.conf ./nitter.conf
 COPY --from=nim /src/nitter/public ./public
 EXPOSE 8080
+RUN adduser -h /src/ -D -s /bin/sh nitter
+USER nitter
 CMD ./nitter

README.md

@@ -4,27 +4,35 @@
 [![Test Matrix](https://github.com/zedeus/nitter/workflows/Docker/badge.svg)](https://github.com/zedeus/nitter/actions/workflows/build-docker.yml)
 [![License](https://img.shields.io/github/license/zedeus/nitter?style=flat)](#license)
 
+> [!NOTE]
+> Running a Nitter instance now requires real accounts, since Twitter removed the previous methods. \
+> This does not affect users. \
+> For instructions on how to obtain session tokens, see [Creating session tokens](https://github.com/zedeus/nitter/wiki/Creating-session-tokens).
+
 A free and open source alternative Twitter front-end focused on privacy and
 performance. \
-Inspired by the [Invidious](https://github.com/iv-org/invidious)
-project.
+Inspired by the [Invidious](https://github.com/iv-org/invidious) project.
 
 - No JavaScript or ads
 - All requests go through the backend, client never talks to Twitter
 - Prevents Twitter from tracking your IP or JavaScript fingerprint
-- Uses Twitter's unofficial API (no rate limits or developer account required)
+- Uses Twitter's unofficial API (no developer account required)
 - Lightweight (for [@nim_lang](https://nitter.net/nim_lang), 60KB vs 784KB from twitter.com)
 - RSS feeds
 - Themes
 - Mobile support (responsive design)
 - AGPLv3 licensed, no proprietary instances permitted
 
-Liberapay: https://liberapay.com/zedeus \
-Patreon: https://patreon.com/nitter \
-BTC: bc1qp7q4qz0fgfvftm5hwz3vy284nue6jedt44kxya \
-ETH: 0x66d84bc3fd031b62857ad18c62f1ba072b011925 \
-LTC: ltc1qhsz5nxw6jw9rdtw9qssjeq2h8hqk2f85rdgpkr \
-XMR: 42hKayRoEAw4D6G6t8mQHPJHQcXqofjFuVfavqKeNMNUZfeJLJAcNU19i1bGdDvcdN6romiSscWGWJCczFLe9RFhM3d1zpL
+<details>
+<summary>Donations</summary>
+Liberapay: https://liberapay.com/zedeus<br>
+Patreon: https://patreon.com/nitter<br>
+BTC: bc1qpqpzjkcpgluhzf7x9yqe7jfe8gpfm5v08mdr55<br>
+ETH: 0x24a0DB59A923B588c7A5EBd0dBDFDD1bCe9c4460<br>
+XMR: 42hKayRoEAw4D6G6t8mQHPJHQcXqofjFuVfavqKeNMNUZfeJLJAcNU19i1bGdDvcdN6romiSscWGWJCczFLe9RFhM3d1zpL<br>
+SOL: ANsyGNXFo6osuFwr1YnUqif2RdoYRhc27WdyQNmmETSW<br>
+ZEC: u1vndfqtzyy6qkzhkapxelel7ams38wmfeccu3fdpy2wkuc4erxyjm8ncjhnyg747x6t0kf0faqhh2hxyplgaum08d2wnj4n7cyu9s6zhxkqw2aef4hgd4s6vh5hpqvfken98rg80kgtgn64ff70djy7s8f839z00hwhuzlcggvefhdlyszkvwy3c7yw623vw3rvar6q6evd3xcvveypt
+</details>
 
 ## Roadmap
 
@@ -42,12 +50,13 @@ maintained by the community.
 
 ## Why?
 
-It's impossible to use Twitter without JavaScript enabled. For privacy-minded
-folks, preventing JavaScript analytics and IP-based tracking is important, but
-apart from using a VPN and uBlock/uMatrix, it's impossible. Despite being behind
-a VPN and using heavy-duty adblockers, you can get accurately tracked with your
-[browser's fingerprint](https://restoreprivacy.com/browser-fingerprinting/),
-[no JavaScript required](https://noscriptfingerprint.com/). This all became
+It's impossible to use Twitter without JavaScript enabled, and as of 2024 you
+need to sign up. For privacy-minded folks, preventing JavaScript analytics and
+IP-based tracking is important, but apart from using a VPN and uBlock/uMatrix,
+it's impossible. Despite being behind a VPN and using heavy-duty adblockers,
+you can get accurately tracked with your [browser's
+fingerprint](https://restoreprivacy.com/browser-fingerprinting/), [no
+JavaScript required](https://noscriptfingerprint.com/). This all became
 particularly important after Twitter [removed the
 ability](https://www.eff.org/deeplinks/2020/04/twitter-removes-privacy-option-and-shows-why-we-need-strong-privacy-laws)
 for users to control whether their data gets sent to advertisers.
@@ -71,19 +80,21 @@ Twitter account.
 
 - libpcre
 - libsass
-- redis
+- redis/valkey
 
 To compile Nitter you need a Nim installation, see
-[nim-lang.org](https://nim-lang.org/install.html) for details. It is possible to
-install it system-wide or in the user directory you create below.
+[nim-lang.org](https://nim-lang.org/install.html) for details. It is possible
+to install it system-wide or in the user directory you create below.
 
 To compile the scss files, you need to install `libsass`. On Ubuntu and Debian,
 you can use `libsass-dev`.
 
-Redis is required for caching and in the future for account info. It should be
-available on most distros as `redis` or `redis-server` (Ubuntu/Debian).
-Running it with the default config is fine, Nitter's default config is set to
-use the default Redis port and localhost.
+Redis is required for caching and in the future for account info. As of 2024
+Redis is no longer open source, so using the fork Valkey is recommended. It
+should be available on most distros as `redis` or `redis-server`
+(Ubuntu/Debian), or `valkey`/`valkey-server`. Running it with the default
+config is fine, Nitter's default config is set to use the default port and
+localhost.
 
 Here's how to create a `nitter` user, clone the repo, and build the project
 along with the scss and md files.
@@ -93,7 +104,7 @@ along with the scss and md files.
 # su nitter
 $ git clone https://github.com/zedeus/nitter
 $ cd nitter
-$ nimble build -d:release
+$ nimble build -d:danger --mm:refc
 $ nimble scss
 $ nimble md
 $ cp nitter.example.conf nitter.conf

config.nims

@@ -7,12 +7,7 @@
 
 # disable annoying warnings
 warning("GcUnsafe2", off)
+warning("HoleEnumConv", off)
 hint("XDeclaredButNotUsed", off)
 hint("XCannotRaiseY", off)
 hint("User", off)
-
-const
-  nimVersion = (major: NimMajor, minor: NimMinor, patch: NimPatch)
-
-when nimVersion >= (1, 6, 0):
-  warning("HoleEnumConv", off)

nitter.example.conf

@@ -23,15 +23,9 @@ redisMaxConnections = 30
 hmacKey = "secretkey"  # random key for cryptographic signing of video urls
 base64Media = false  # use base64 encoding for proxied media urls
 enableRSS = true  # set this to false to disable RSS feeds
-enableDebug = false  # enable request logs and debug endpoints (/.tokens)
+enableDebug = false  # enable request logs and debug endpoints (/.sessions)
 proxy = ""  # http/https url, SOCKS proxies are not supported
 proxyAuth = ""
-tokenCount = 10
-# minimum amount of usable tokens. tokens are used to authorize API requests,
-# but they expire after ~1 hour, and have a limit of 500 requests per endpoint.
-# the limits reset every 15 minutes, and the pool is filled up so there's
-# always at least `tokenCount` usable tokens. only increase this if you receive
-# major bursts all the time and don't have a rate limiting setup via e.g. nginx
 
 # Change default preferences here, see src/prefs_impl.nim for a complete list
 [Preferences]

nitter.nimble

@@ -14,16 +14,16 @@ requires "nim >= 1.6.10"
 requires "jester#baca3f"
 requires "karax#5cf360c"
 requires "sass#7dfdd03"
-requires "nimcrypto#4014ef9"
+requires "nimcrypto#a079df9"
 requires "markdown#158efe3"
 requires "packedjson#9e6fbb6"
 requires "supersnappy#6c94198"
 requires "redpool#8b7c1db"
 requires "https://github.com/zedeus/redis#d0a0e6f"
 requires "zippy#ca5989a"
 requires "flatty#e668085"
-requires "jsony#ea811be"
-
+requires "jsony#1de1f08"
+requires "oauth#b8c163b"
 
 # Tasks
 

public/js/hls.light.min.js

@@ -4,15 +4,15 @@ Nitter is a free and open source alternative Twitter front-end focused on
 privacy and performance. The source is available on GitHub at
 <https://github.com/zedeus/nitter>
 
-* No JavaScript or ads
-* All requests go through the backend, client never talks to Twitter
-* Prevents Twitter from tracking your IP or JavaScript fingerprint
-* Uses Twitter's unofficial API (no rate limits or developer account required)
-* Lightweight (for [@nim_lang](/nim_lang), 60KB vs 784KB from twitter.com)
-* RSS feeds
-* Themes
-* Mobile support (responsive design)
-* AGPLv3 licensed, no proprietary instances permitted
+- No JavaScript or ads
+- All requests go through the backend, client never talks to Twitter
+- Prevents Twitter from tracking your IP or JavaScript fingerprint
+- Uses Twitter's unofficial API (no developer account required)
+- Lightweight (for [@nim_lang](/nim_lang), 60KB vs 784KB from twitter.com)
+- RSS feeds
+- Themes
+- Mobile support (responsive design)
+- AGPLv3 licensed, no proprietary instances permitted
 
 Nitter's GitHub wiki contains
 [instances](https://github.com/zedeus/nitter/wiki/Instances) and
@@ -21,12 +21,13 @@ maintained by the community.
 
 ## Why use Nitter?
 
-It's impossible to use Twitter without JavaScript enabled. For privacy-minded
-folks, preventing JavaScript analytics and IP-based tracking is important, but
-apart from using a VPN and uBlock/uMatrix, it's impossible. Despite being behind
-a VPN and using heavy-duty adblockers, you can get accurately tracked with your
-[browser's fingerprint](https://restoreprivacy.com/browser-fingerprinting/),
-[no JavaScript required](https://noscriptfingerprint.com/). This all became
+It's impossible to use Twitter without JavaScript enabled, and as of 2024 you
+need to sign up. For privacy-minded folks, preventing JavaScript analytics and
+IP-based tracking is important, but apart from using a VPN and uBlock/uMatrix,
+it's impossible. Despite being behind a VPN and using heavy-duty adblockers,
+you can get accurately tracked with your [browser's
+fingerprint](https://restoreprivacy.com/browser-fingerprinting/), [no
+JavaScript required](https://noscriptfingerprint.com/). This all became
 particularly important after Twitter [removed the
 ability](https://www.eff.org/deeplinks/2020/04/twitter-removes-privacy-option-and-shows-why-we-need-strong-privacy-laws)
 for users to control whether their data gets sent to advertisers.
@@ -42,12 +43,13 @@ Twitter account.
 
 ## Donating
 
-Liberapay: <https://liberapay.com/zedeus> \
-Patreon: <https://patreon.com/nitter> \
-BTC: bc1qp7q4qz0fgfvftm5hwz3vy284nue6jedt44kxya \
-ETH: 0x66d84bc3fd031b62857ad18c62f1ba072b011925 \
-LTC: ltc1qhsz5nxw6jw9rdtw9qssjeq2h8hqk2f85rdgpkr \
-XMR: 42hKayRoEAw4D6G6t8mQHPJHQcXqofjFuVfavqKeNMNUZfeJLJAcNU19i1bGdDvcdN6romiSscWGWJCczFLe9RFhM3d1zpL
+Liberapay: https://liberapay.com/zedeus \
+Patreon: https://patreon.com/nitter \
+BTC: bc1qpqpzjkcpgluhzf7x9yqe7jfe8gpfm5v08mdr55 \
+ETH: 0x24a0DB59A923B588c7A5EBd0dBDFDD1bCe9c4460 \
+XMR: 42hKayRoEAw4D6G6t8mQHPJHQcXqofjFuVfavqKeNMNUZfeJLJAcNU19i1bGdDvcdN6romiSscWGWJCczFLe9RFhM3d1zpL \
+SOL: ANsyGNXFo6osuFwr1YnUqif2RdoYRhc27WdyQNmmETSW \
+ZEC: u1vndfqtzyy6qkzhkapxelel7ams38wmfeccu3fdpy2wkuc4erxyjm8ncjhnyg747x6t0kf0faqhh2hxyplgaum08d2wnj4n7cyu9s6zhxkqw2aef4hgd4s6vh5hpqvfken98rg80kgtgn64ff70djy7s8f839z00hwhuzlcggvefhdlyszkvwy3c7yw623vw3rvar6q6evd3xcvveypt
 
 ## Contact